Oct 31 03:12:48.899275 ip-10-0-141-153 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 03:12:48.899286 ip-10-0-141-153 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 03:12:48.899293 ip-10-0-141-153 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 03:12:48.899538 ip-10-0-141-153 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 03:12:58.932851 ip-10-0-141-153 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 03:12:58.932868 ip-10-0-141-153 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 80ce1c9154b2459c925802af9145d80e -- Oct 31 03:15:03.381513 ip-10-0-141-153 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:15:03.717471 ip-10-0-141-153 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:15:03.717471 ip-10-0-141-153 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:15:03.717471 ip-10-0-141-153 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:15:03.717471 ip-10-0-141-153 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:15:03.722603 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722514 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.722603 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722600 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.722603 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722604 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.722603 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722607 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.722603 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722610 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.722603 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722612 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722617 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722620 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722622 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722625 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722627 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722630 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722633 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722635 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722637 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722640 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722642 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722645 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722647 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722649 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722654 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722658 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722663 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.722823 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722666 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722669 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722671 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722674 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722676 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722679 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722681 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722691 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722694 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722697 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722700 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722703 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722705 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722708 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722711 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722714 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722717 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722720 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722722 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722725 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.723399 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722727 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722730 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722732 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722735 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722738 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722740 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722744 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722747 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722750 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722752 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722755 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722757 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722760 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722762 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722765 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722767 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722770 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722772 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722774 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.723904 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722778 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722782 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722785 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722788 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722791 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722794 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722796 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722800 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722802 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722805 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722807 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722810 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722813 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722815 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722818 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722821 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722824 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722827 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722830 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722832 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.724371 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722836 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722839 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722841 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722844 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722846 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722849 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722852 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722854 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722857 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722859 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722862 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722864 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722867 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722870 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722872 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722875 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722878 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722880 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722884 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.724837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722887 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722889 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722892 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.722896 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723682 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723689 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723692 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723695 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723698 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723700 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723703 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723706 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723708 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723711 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723714 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723717 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723720 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723722 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723725 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.725296 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723728 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723730 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723733 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723735 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723738 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723740 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723743 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723745 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723748 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723751 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723754 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723756 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723760 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723764 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723767 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723770 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723773 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723775 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723778 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723780 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.725760 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723782 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723785 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723788 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723791 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723793 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723795 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723798 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723800 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723803 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723806 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723809 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723811 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723814 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723817 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723819 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723822 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723824 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723827 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723830 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723833 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.726263 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723835 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723838 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723840 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723843 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723845 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723849 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723854 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723857 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723859 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723862 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723865 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723867 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723870 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723872 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723875 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723877 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723880 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723882 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723885 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.726778 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723888 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723891 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723894 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723897 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723900 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723902 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723904 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723907 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723910 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723912 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723915 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723918 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723920 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723923 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723925 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723928 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723930 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723933 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723936 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723938 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.727253 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723941 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723944 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723946 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723948 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723951 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723953 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723956 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723958 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723961 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723963 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723967 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.723970 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724479 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724485 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724488 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724491 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724494 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724497 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724500 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724502 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.727737 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724505 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724508 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724510 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724513 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724515 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724518 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724521 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724523 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724526 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724528 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724532 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724535 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724538 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724541 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724544 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724546 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724549 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724551 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724554 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.728221 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724556 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724559 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724561 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724564 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724566 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724569 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724572 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724574 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724577 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724580 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724582 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724584 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724587 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724589 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724592 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724594 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724597 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724600 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724602 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.728680 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724604 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724607 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724609 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724611 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724614 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724616 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724619 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724621 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724623 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724626 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724630 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724634 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724638 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724641 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724644 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724647 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724649 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724651 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724654 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.729125 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724656 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724659 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724662 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724666 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724668 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724671 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724673 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724676 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724678 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724680 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724683 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724685 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724688 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724690 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724692 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724695 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724697 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724700 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724703 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724705 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.729586 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724707 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724710 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724712 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724714 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724717 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724719 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724722 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724724 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724727 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724729 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724731 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724734 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724736 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724741 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724743 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724746 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724749 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724751 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724754 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724756 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.724759 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.730051 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725627 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725637 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725645 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725649 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725654 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725657 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725661 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725665 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725668 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725671 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725678 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725682 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725685 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725688 2579 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725690 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725693 2579 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725696 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725699 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725706 2579 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725709 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725712 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725715 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725718 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725727 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:15:03.730575 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725731 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725734 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725737 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725740 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725743 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725746 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725749 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725752 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725756 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725759 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725762 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725765 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725768 2579 flags.go:64] FLAG: --enable-server="true" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725771 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725775 2579 flags.go:64] FLAG: --event-burst="100" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725778 2579 flags.go:64] FLAG: --event-qps="50" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725781 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725784 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725787 2579 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725791 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725794 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725797 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725800 2579 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725803 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725806 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:15:03.731135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725809 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725813 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725816 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725819 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725822 2579 flags.go:64] FLAG: --feature-gates="" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725826 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725829 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725833 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725836 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725839 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725842 2579 flags.go:64] FLAG: --help="false" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725845 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725850 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725853 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725856 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725859 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725863 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725866 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725869 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725872 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725875 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725878 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725881 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725883 2579 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:15:03.731790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725886 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725889 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725892 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725895 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725898 2579 flags.go:64] FLAG: --lock-file="" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725900 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725903 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725906 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725912 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725917 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725920 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725923 2579 flags.go:64] FLAG: --logging-format="text" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725926 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725929 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725932 2579 flags.go:64] FLAG: --manifest-url="" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725937 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725941 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725944 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725948 2579 flags.go:64] FLAG: --max-pods="110" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725963 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725967 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725971 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725974 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725978 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725980 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:15:03.732372 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725983 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725991 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725994 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.725997 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726000 2579 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726003 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726006 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726009 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726012 2579 flags.go:64] FLAG: --port="10250" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726015 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726018 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0047b1328d2b4b254" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726021 2579 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726024 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726027 2579 flags.go:64] FLAG: --register-node="true" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726029 2579 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726033 2579 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726039 2579 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726041 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726044 2579 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726048 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726051 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726054 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726058 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726061 2579 flags.go:64] FLAG: --runonce="false" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726064 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726067 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726070 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:15:03.732973 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726073 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726075 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726079 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726082 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726085 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726088 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726091 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726094 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726097 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726100 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726102 2579 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726105 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726110 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726113 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726116 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726120 2579 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726122 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726125 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726128 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726131 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726134 2579 flags.go:64] FLAG: --v="2" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726139 2579 flags.go:64] FLAG: --version="false" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726144 2579 flags.go:64] FLAG: --vmodule="" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726148 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726151 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:15:03.733636 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726299 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726304 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726308 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726311 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726314 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726316 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726319 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726322 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726324 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726327 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726330 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726333 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726335 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726338 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726340 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726343 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726345 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726347 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726350 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726352 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.734230 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726355 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726357 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726360 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726363 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726365 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726367 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726370 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726372 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726376 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726379 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726381 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726384 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726386 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726389 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726391 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726394 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726397 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726400 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726402 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726405 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.734701 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726407 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726410 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726414 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726418 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726422 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726426 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726429 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726432 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726434 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726437 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726439 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726442 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726444 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726447 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726449 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726452 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726454 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726457 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726459 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.735173 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726462 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726465 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726468 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726470 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726473 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726475 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726478 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726480 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726483 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726485 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726488 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726490 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726493 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726495 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726498 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726501 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726503 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726506 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726509 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726511 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.735673 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726514 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726516 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726518 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726521 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726523 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726526 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726528 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726531 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726533 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726536 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726538 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726541 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726544 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726547 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726550 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726552 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726555 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726557 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726560 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726563 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.736649 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726566 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726568 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726571 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726573 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726576 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726578 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726581 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.726607 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726642 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726646 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726650 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726653 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726656 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726658 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726661 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726663 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.737132 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726666 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726668 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726670 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726673 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726675 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726678 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726682 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726684 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726687 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726689 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726693 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726696 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726698 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726701 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726703 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726706 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726708 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726710 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726713 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.737536 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726715 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726718 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726720 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726723 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726725 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726728 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726730 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726746 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726750 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726754 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726757 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726759 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726762 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726764 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726767 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726770 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726772 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726775 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726777 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.737995 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726783 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726787 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726790 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726794 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726797 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726801 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726803 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726806 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726808 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726811 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726813 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726816 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726818 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726821 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726823 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726826 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726828 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726831 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726833 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.738469 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726836 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726838 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726840 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726860 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726863 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726866 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726868 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726871 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726873 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726876 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726878 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726880 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726883 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726887 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726890 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726892 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726894 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726897 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726900 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726902 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.738910 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726905 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726908 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726910 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726912 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726915 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726918 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726920 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726923 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726925 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726928 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726930 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726933 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726936 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726938 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726940 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726943 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726947 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726949 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726952 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.739398 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726954 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.726957 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.733841 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.733857 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.733873 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.733878 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733933 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733938 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733941 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733944 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733947 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733950 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733953 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733955 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733958 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733960 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733963 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733965 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733968 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733971 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.739844 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733973 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733976 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733979 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733981 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733984 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733986 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733989 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733993 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.733997 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734000 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734003 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734005 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734008 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734011 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734013 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734017 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734020 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734025 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.740445 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734028 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734030 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734033 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734035 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734038 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734040 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734043 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734045 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734048 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734051 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734053 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734056 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734059 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734062 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734064 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734067 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734069 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734074 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.740962 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734077 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734079 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734082 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734084 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734087 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734090 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734092 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734095 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734097 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734100 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734104 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734107 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734110 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734112 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734114 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734117 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734119 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734121 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734124 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734126 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.741722 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734128 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734131 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734133 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734135 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734138 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734141 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734145 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734149 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734151 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734154 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734156 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734159 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734161 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734164 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734166 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734169 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734171 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734174 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734176 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.742372 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734179 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734181 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734184 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734187 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734190 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734193 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734195 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734215 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734219 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734223 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734226 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734228 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734231 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734234 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734236 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.742816 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.734263 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734409 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734413 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734416 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734418 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734421 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734423 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734426 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734428 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734431 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734433 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734436 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734438 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734441 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734443 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734446 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734448 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734451 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734454 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.743181 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734456 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734459 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734461 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734466 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734469 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734472 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734475 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734478 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734480 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734483 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734485 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734488 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734490 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734492 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734495 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734497 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734500 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734502 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734505 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.744000 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734507 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734509 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734512 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734514 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734517 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734519 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734522 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734524 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734527 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734529 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734531 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734534 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734536 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734539 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734541 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734544 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734546 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734549 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734551 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734554 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.744571 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734556 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734559 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734561 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734564 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734566 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734569 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734571 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734573 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734576 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734578 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734581 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734583 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734585 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734589 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734593 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734596 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734598 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734601 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734604 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.745042 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734607 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734609 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734611 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734614 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734617 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734619 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734622 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734624 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734626 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734629 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734631 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734634 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734637 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734639 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734642 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734644 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734646 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734649 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734651 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734653 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.745521 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734656 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734658 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734661 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734663 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734666 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734668 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734670 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734673 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734675 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734678 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.734700 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734731 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734735 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734738 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734740 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734742 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:15:03.745992 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734745 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734748 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734750 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734752 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734755 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734757 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734760 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734762 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734765 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734768 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734770 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734773 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734775 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734777 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734780 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734782 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734785 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734787 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734789 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734792 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:15:03.746393 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734794 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734797 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734799 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734801 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734804 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734806 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734813 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734816 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734820 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734822 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734825 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734828 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734832 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734834 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734837 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734840 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734842 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734845 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:15:03.746983 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734847 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734850 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734853 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734855 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734858 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734860 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734863 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734865 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734868 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734870 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734872 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734876 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734878 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734881 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734883 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734886 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734888 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734891 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734893 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734897 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:15:03.747519 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734899 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734901 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734905 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734908 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734910 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734912 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734915 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734917 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734920 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734922 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734925 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734928 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734930 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734932 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734935 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734938 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734940 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734942 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734945 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:15:03.748093 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734947 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734950 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734952 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734956 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734960 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734964 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734966 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734969 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734972 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734977 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734979 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734983 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734985 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734988 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734991 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734994 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.734996 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.735000 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:15:03.748690 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.735002 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.735005 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.735007 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:03.735010 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.735059 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.736895 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.737651 2579 server.go:1014] "Starting client certificate rotation" Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.737749 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.737784 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.739748 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:15:03.749165 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.744352 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:15:03.755319 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.755303 2579 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:15:03.758240 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.758228 2579 log.go:25] "Validated CRI v1 image API" Oct 31 03:15:03.759076 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.759040 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:15:03.761309 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.761274 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bce90099-8273-4ef9-8653-de7b6659b793:/dev/nvme0n1p3 e5ebff43-e782-4b35-890a-9b8ced84f7ef:/dev/nvme0n1p4] Oct 31 03:15:03.761354 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.761310 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 03:15:03.764316 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.764299 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:15:03.766263 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.766135 2579 manager.go:217] Machine: {Timestamp:2026-10-31 03:15:03.764971318 +0000 UTC m=+0.283335520 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101868 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d870904cd2693df0b75f66f996a52 SystemUUID:ec2d8709-04cd-2693-df0b-75f66f996a52 BootID:80ce1c91-54b2-459c-9258-02af9145d80e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:ee:83:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:ee:83:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:27:6f:17:e9:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:15:03.766263 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.766262 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:15:03.766375 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.766340 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:15:03.769162 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.769140 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:15:03.769313 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.769164 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-153.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:15:03.769362 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.769315 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:15:03.769362 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.769323 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:15:03.769362 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.769355 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:15:03.769452 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.769373 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:15:03.770041 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.770031 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:15:03.773500 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.773487 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:15:03.773625 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.773616 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:15:03.773662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.773654 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:15:03.773692 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.773670 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:15:03.773692 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.773682 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:15:03.773692 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.773691 2579 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:15:03.773782 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.773699 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:15:03.774127 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.774115 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:15:03.775896 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.775881 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:15:03.780982 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.780967 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:15:03.781715 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781703 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:15:03.781769 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781736 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:15:03.781891 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781881 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:15:03.781937 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781896 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:15:03.781937 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781902 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:15:03.781937 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781908 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:15:03.781937 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781914 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:15:03.781937 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.781920 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:15:03.782325 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.782315 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:15:03.782356 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.782326 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:15:03.782356 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.782336 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:15:03.782356 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.782343 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:15:03.782518 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.782361 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:15:03.782518 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.782371 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:15:03.783259 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.783246 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:15:03.783294 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.783265 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:15:03.784729 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.784699 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zh2vd" Oct 31 03:15:03.788432 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.788414 2579 server.go:1263] "Started kubelet" Oct 31 03:15:03.788528 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.788426 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-153.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 03:15:03.788528 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.788469 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:15:03.788871 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.788626 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:15:03.788871 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.788706 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:15:03.789349 ip-10-0-141-153 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:15:03.790350 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.790339 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:15:03.790623 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.790607 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zh2vd" Oct 31 03:15:03.792671 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.792655 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:15:03.793744 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.793726 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 03:15:03.795782 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.795759 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 03:15:03.795839 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.795789 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-153.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 03:15:03.796493 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.795713 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-153.us-east-2.compute.internal.18e37eceeaa0fc1a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-153.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-153.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 03:15:03.78837097 +0000 UTC m=+0.306735171,LastTimestamp:2026-10-31 03:15:03.78837097 +0000 UTC m=+0.306735171,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-153.us-east-2.compute.internal,}" Oct 31 03:15:03.796623 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.796598 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:15:03.796995 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.796975 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:15:03.797083 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.797064 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:03.797482 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.797463 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:15:03.797482 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.797478 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:15:03.797482 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.797484 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:15:03.797638 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.797563 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:15:03.797638 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.797575 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:15:03.797971 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.797954 2579 factory.go:55] Registering systemd factory Oct 31 03:15:03.797971 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.797971 2579 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:15:03.798232 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.798217 2579 factory.go:153] Registering CRI-O factory Oct 31 03:15:03.798292 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.798235 2579 factory.go:223] Registration of the crio container factory successfully Oct 31 03:15:03.798338 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.798307 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:15:03.798338 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.798323 2579 factory.go:103] Registering Raw factory Oct 31 03:15:03.798338 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.798337 2579 manager.go:1199] Started watching for new ooms in manager Oct 31 03:15:03.799171 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.799157 2579 manager.go:319] Starting recovery of all containers Oct 31 03:15:03.801649 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.801561 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:15:03.807487 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.807460 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-153.us-east-2.compute.internal\" not found" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:03.809900 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.809881 2579 manager.go:324] Recovery completed Oct 31 03:15:03.815115 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.815097 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:15:03.817434 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.817419 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:15:03.817483 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.817448 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:15:03.817483 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.817458 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:15:03.818000 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.817987 2579 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:15:03.818000 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.817996 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:15:03.818108 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.817997 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:15:03.818108 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.818045 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:15:03.818707 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.818694 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:15:03.818772 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.818725 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:15:03.818772 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.818740 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:15:03.819989 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.819977 2579 policy_none.go:50] "Start" Oct 31 03:15:03.820031 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.819993 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:15:03.820031 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.820002 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:15:03.821889 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.821878 2579 policy_none.go:44] "Start" Oct 31 03:15:03.855757 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.855728 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:15:03.858050 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.858035 2579 manager.go:341] "Starting Device Plugin manager" Oct 31 03:15:03.858153 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.858063 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:15:03.858153 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.858072 2579 server.go:87] "Starting device plugin registration server" Oct 31 03:15:03.858357 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.858343 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:15:03.858413 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.858357 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:15:03.858459 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.858437 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:15:03.858535 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.858521 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:15:03.858535 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.858534 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:15:03.859051 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.859028 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:15:03.859136 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.859084 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:03.947603 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.947565 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:15:03.948934 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.948900 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:15:03.949036 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.948941 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:15:03.949036 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.948970 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:15:03.949036 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.949013 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 03:15:03.952845 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.952829 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:15:03.959159 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.959144 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:15:03.959937 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.959922 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:15:03.959986 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.959953 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:15:03.959986 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.959962 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:15:03.959986 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.959985 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:03.967945 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:03.967908 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:03.967945 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.967930 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-153.us-east-2.compute.internal\": node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:03.978326 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:03.978309 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.049782 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.049735 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal"] Oct 31 03:15:04.049944 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.049822 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:15:04.050807 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.050783 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:15:04.050923 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.050824 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:15:04.050923 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.050836 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:15:04.053036 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053023 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:15:04.053181 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.053252 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053228 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:15:04.053744 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053728 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:15:04.053846 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053758 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:15:04.053846 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053770 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:15:04.053846 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053732 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:15:04.053846 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053832 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:15:04.053846 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.053844 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:15:04.055977 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.055962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.056062 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.055991 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:15:04.056659 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.056641 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:15:04.056762 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.056669 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:15:04.056762 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.056679 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:15:04.078379 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.078357 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.080627 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.080609 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.082886 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.082870 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.098635 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.098614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.098719 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.098644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.179192 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.179029 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.199338 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.199313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.199423 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.199344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.199423 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.199365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a5e637a257f84120786f81c645c34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"057a5e637a257f84120786f81c645c34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.199423 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.199405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.199537 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.199420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.279783 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.279713 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.300060 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.300028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a5e637a257f84120786f81c645c34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"057a5e637a257f84120786f81c645c34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.300121 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.300078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a5e637a257f84120786f81c645c34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"057a5e637a257f84120786f81c645c34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.380236 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.380190 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.383368 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.383350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.385507 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.385493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:04.481157 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.481117 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.581705 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.581632 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.682107 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.682078 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.737553 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.737531 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 03:15:04.738016 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.737715 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:15:04.738016 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.737722 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:15:04.783028 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.782994 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.798008 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.797962 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:10:04 +0000 UTC" deadline="2028-04-19 05:27:52.148990152 +0000 UTC" Oct 31 03:15:04.798008 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.798003 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12866h12m47.350990681s" Oct 31 03:15:04.883403 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.883313 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:04.897295 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:04.897273 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:15:04.984306 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:04.984276 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:15:05.030432 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.030399 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:15:05.075520 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:05.075474 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a0fa3bb14222e5d3d01c4ed0d8f38f.slice/crio-41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200 WatchSource:0}: Error finding container 41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200: Status 404 returned error can't find the container with id 41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200 Oct 31 03:15:05.075956 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:05.075939 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057a5e637a257f84120786f81c645c34.slice/crio-0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a WatchSource:0}: Error finding container 0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a: Status 404 returned error can't find the container with id 0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a Oct 31 03:15:05.080601 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.080587 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:15:05.097412 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.097395 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:05.108528 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.108508 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:15:05.109346 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.109332 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:15:05.118900 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.118888 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:15:05.659992 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.659963 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:15:05.775648 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.775611 2579 apiserver.go:52] "Watching apiserver" Oct 31 03:15:05.796717 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.796686 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:15:05.798422 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.797087 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 03:15:05.798422 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.797802 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-flc2b","openshift-ovn-kubernetes/ovnkube-node-8z9f4","kube-system/konnectivity-agent-t5fpc","openshift-multus/multus-additional-cni-plugins-rrqw6","openshift-network-diagnostics/network-check-target-dlvjw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6","openshift-cluster-node-tuning-operator/tuned-gfmjc","openshift-multus/multus-gqnj2","kube-system/global-pull-secret-syncer-9cczw","openshift-dns/node-resolver-dbm7r","openshift-image-registry/node-ca-xwkgw","openshift-network-operator/iptables-alerter-ghj2p"] Oct 31 03:15:05.800964 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.800913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:05.801080 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:05.800991 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:05.803008 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.802982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:05.806069 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.806049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:05.806173 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:05.806114 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:05.806605 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.806544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:05.808127 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.808102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.809972 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.809606 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:15:05.809972 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.809630 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:15:05.809972 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.809695 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:15:05.809972 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.809773 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fn5cc\"" Oct 31 03:15:05.811036 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.811004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:05.814064 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.814038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.814734 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.814717 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:15:05.815048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.815029 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:15:05.815290 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.815273 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:15:05.817059 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.817033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.818678 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.818655 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:15:05.819551 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.819532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:05.823305 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.822391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.824652 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.824631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.827377 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.827357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:05.827508 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:05.827442 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:05.830574 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.830555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:05.838099 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.838079 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mr4vl" Oct 31 03:15:05.849478 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.849455 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mr4vl" Oct 31 03:15:05.887191 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.887155 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:15:05.898377 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.898352 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:15:05.907660 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.907635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-kubernetes\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.907785 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.907708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-hostroot\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.907785 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.907763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsljf\" (UniqueName: \"kubernetes.io/projected/196bf414-2c29-40a0-a957-f94cadaa7ff2-kube-api-access-vsljf\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:05.907909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.907850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-netns\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.907909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.907899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f454b995-5cf2-461d-b6f1-932bd4a45110-konnectivity-ca\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:05.908022 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.907930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2mm\" (UniqueName: \"kubernetes.io/projected/f771de0f-1ffa-4de2-823d-40d0a95eed5c-kube-api-access-8t2mm\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.908079 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4p9n\" (UniqueName: \"kubernetes.io/projected/453eddca-0701-4325-b5e2-12b4e3e78565-kube-api-access-k4p9n\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:05.908138 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-device-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.908138 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-etc-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.908245 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.908327 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.908327 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-socket-dir-parent\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.908430 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvzf2\" (UniqueName: \"kubernetes.io/projected/d9c43876-cf34-43bb-b641-01e1b6a4b143-kube-api-access-xvzf2\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:05.908430 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.908430 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysconfig\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.908588 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-var-lib-kubelet\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.908588 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-tmp\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.908588 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-multus-certs\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.908737 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196bf414-2c29-40a0-a957-f94cadaa7ff2-host\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:05.908796 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-lib-modules\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.908796 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-cnibin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.908887 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-k8s-cni-cncf-io\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.908887 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-kubelet\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.908887 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-systemd-units\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909045 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-slash\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909045 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-env-overrides\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909045 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.908995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-sys\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.909166 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-system-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.909166 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjxn\" (UniqueName: \"kubernetes.io/projected/3387800d-492d-4f65-b3d8-4bbccd906d03-kube-api-access-pnjxn\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:05.909166 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovn-node-metrics-cert\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909166 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.909166 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-kubelet-config\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-kubelet\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-ovn\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-config\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-modprobe-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvlv\" (UniqueName: \"kubernetes.io/projected/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kube-api-access-dwvlv\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-cni-binary-copy\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-system-cni-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.909396 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-bin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-netd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cnibin\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-host\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-etc-tuned\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-log-socket\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-systemd\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-run\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-socket-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-registration-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/196bf414-2c29-40a0-a957-f94cadaa7ff2-serviceca\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/453eddca-0701-4325-b5e2-12b4e3e78565-host-slash\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:05.909838 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-netns\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-conf-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3387800d-492d-4f65-b3d8-4bbccd906d03-tmp-dir\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-node-log\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bmk\" (UniqueName: \"kubernetes.io/projected/5ab0c10c-8503-43e7-8892-894c3c0650d4-kube-api-access-72bmk\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ftdc\" (UniqueName: \"kubernetes.io/projected/f07e2583-7e4e-49ef-8980-e76733289e46-kube-api-access-8ftdc\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.909994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-sys-fs\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-multus\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9db\" (UniqueName: \"kubernetes.io/projected/8e6a1947-49a2-44d8-9db3-5004938650e6-kube-api-access-sk9db\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-var-lib-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-bin\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-script-lib\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-conf\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-dbus\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:05.910491 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3387800d-492d-4f65-b3d8-4bbccd906d03-hosts-file\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-systemd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-etc-kubernetes\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910430 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f454b995-5cf2-461d-b6f1-932bd4a45110-agent-certs\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/453eddca-0701-4325-b5e2-12b4e3e78565-iptables-alerter-script\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:05.911196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.910484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-daemon-config\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:05.916448 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.916423 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:15:05.916542 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.916532 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:15:05.916703 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.916684 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:15:05.917024 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.917007 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:15:05.917103 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.917070 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:15:05.917295 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.917217 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:15:05.919041 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.919023 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:15:05.919125 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.919083 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:15:05.919183 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.919128 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:15:05.919970 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.919951 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:15:05.920062 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.919996 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:15:05.920062 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.920045 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.920869 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.920931 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.920944 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmjq8\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921007 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b55zh\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921040 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nklpg\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921068 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921084 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921134 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qg7r7\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921143 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkjgm\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921178 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921208 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5s7vx\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921220 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rtvr4\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921235 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921312 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:15:05.921371 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921009 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:15:05.922155 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.921432 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:15:05.922403 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.922371 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6lm2\"" Oct 31 03:15:05.922491 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:05.922479 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:05.922547 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:05.922495 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:05.922547 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:05.922509 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kfm56 for pod openshift-network-diagnostics/network-check-target-dlvjw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:05.922647 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:05.922562 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56 podName:6b1ec6c2-6309-4e90-ae0b-b85aefc8c637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:06.422547404 +0000 UTC m=+2.940911594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kfm56" (UniqueName: "kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56") pod "network-check-target-dlvjw" (UID: "6b1ec6c2-6309-4e90-ae0b-b85aefc8c637") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:05.956482 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.956422 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"057a5e637a257f84120786f81c645c34","Type":"ContainerStarted","Data":"0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a"} Oct 31 03:15:05.958442 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:05.958416 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"b6a0fa3bb14222e5d3d01c4ed0d8f38f","Type":"ContainerStarted","Data":"41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200"} Oct 31 03:15:06.011485 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-kubernetes\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-hostroot\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsljf\" (UniqueName: \"kubernetes.io/projected/196bf414-2c29-40a0-a957-f94cadaa7ff2-kube-api-access-vsljf\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-netns\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f454b995-5cf2-461d-b6f1-932bd4a45110-konnectivity-ca\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2mm\" (UniqueName: \"kubernetes.io/projected/f771de0f-1ffa-4de2-823d-40d0a95eed5c-kube-api-access-8t2mm\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011597 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-kubernetes\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p9n\" (UniqueName: \"kubernetes.io/projected/453eddca-0701-4325-b5e2-12b4e3e78565-kube-api-access-k4p9n\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:06.011662 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-device-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-etc-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-socket-dir-parent\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvzf2\" (UniqueName: \"kubernetes.io/projected/d9c43876-cf34-43bb-b641-01e1b6a4b143-kube-api-access-xvzf2\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysconfig\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-var-lib-kubelet\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-tmp\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-multus-certs\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196bf414-2c29-40a0-a957-f94cadaa7ff2-host\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.011978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-lib-modules\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-cnibin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-k8s-cni-cncf-io\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-kubelet\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.012082 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-systemd-units\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-slash\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-env-overrides\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012135 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f454b995-5cf2-461d-b6f1-932bd4a45110-konnectivity-ca\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-sys\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-system-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-hostroot\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjxn\" (UniqueName: \"kubernetes.io/projected/3387800d-492d-4f65-b3d8-4bbccd906d03-kube-api-access-pnjxn\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovn-node-metrics-cert\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-kubelet-config\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-kubelet\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-ovn\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-config\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012430 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-netns\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-modprobe-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvlv\" (UniqueName: \"kubernetes.io/projected/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kube-api-access-dwvlv\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.012909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-cni-binary-copy\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-system-cni-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-bin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-netd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cnibin\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-host\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-etc-tuned\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-log-socket\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-systemd\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-run\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-socket-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-registration-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.013817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/196bf414-2c29-40a0-a957-f94cadaa7ff2-serviceca\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.012997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/453eddca-0701-4325-b5e2-12b4e3e78565-host-slash\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013065 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/453eddca-0701-4325-b5e2-12b4e3e78565-host-slash\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-netns\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-conf-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3387800d-492d-4f65-b3d8-4bbccd906d03-tmp-dir\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-node-log\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bmk\" (UniqueName: \"kubernetes.io/projected/5ab0c10c-8503-43e7-8892-894c3c0650d4-kube-api-access-72bmk\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftdc\" (UniqueName: \"kubernetes.io/projected/f07e2583-7e4e-49ef-8980-e76733289e46-kube-api-access-8ftdc\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-sys-fs\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-multus\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9db\" (UniqueName: \"kubernetes.io/projected/8e6a1947-49a2-44d8-9db3-5004938650e6-kube-api-access-sk9db\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-var-lib-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-bin\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-script-lib\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-conf\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.014620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-dbus\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3387800d-492d-4f65-b3d8-4bbccd906d03-hosts-file\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-systemd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-device-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-etc-kubernetes\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f454b995-5cf2-461d-b6f1-932bd4a45110-agent-certs\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/453eddca-0701-4325-b5e2-12b4e3e78565-iptables-alerter-script\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-daemon-config\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013821 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysconfig\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013871 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-var-lib-kubelet\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-conf-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013937 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-node-log\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.013659 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-etc-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014183 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:15:06.015156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014249 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014417 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-sys-fs\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-multus\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014602 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3387800d-492d-4f65-b3d8-4bbccd906d03-tmp-dir\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-systemd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-etc-kubernetes\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-var-lib-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.014739 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-bin\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.014746 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.014791 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs podName:d9c43876-cf34-43bb-b641-01e1b6a4b143 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:06.514778149 +0000 UTC m=+3.033142354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs") pod "network-metrics-daemon-flc2b" (UID: "d9c43876-cf34-43bb-b641-01e1b6a4b143") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-multus-certs\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015334 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196bf414-2c29-40a0-a957-f94cadaa7ff2-host\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015426 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-lib-modules\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015483 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-cnibin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015528 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-k8s-cni-cncf-io\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-kubelet\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015622 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-systemd-units\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016048 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-script-lib\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015664 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-slash\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3387800d-492d-4f65-b3d8-4bbccd906d03-hosts-file\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015801 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-conf\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.015851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-dbus\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.015911 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.015955 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret podName:73113784-937f-492c-9ac1-1a98c59d43e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:06.515939304 +0000 UTC m=+3.034303513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret") pod "global-pull-secret-syncer-9cczw" (UID: "73113784-937f-492c-9ac1-1a98c59d43e6") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016064 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-env-overrides\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016132 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-sys\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016132 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-system-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/453eddca-0701-4325-b5e2-12b4e3e78565-iptables-alerter-script\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016382 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-netd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016390 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-socket-dir-parent\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016478 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-modprobe-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016538 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016589 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-log-socket\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.016985 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016590 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-daemon-config\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-registration-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.016700 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-socket-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-ovn\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017154 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-systemd\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-netns\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017202 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-run\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017327 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-kubelet\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-bin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017412 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-kubelet-config\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017425 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017462 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-system-cni-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017464 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cnibin\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.017820 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017729 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.018560 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.017833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-config\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.018560 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.018083 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/196bf414-2c29-40a0-a957-f94cadaa7ff2-serviceca\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:06.018560 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.018120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.018560 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.018551 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-cni-binary-copy\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.018761 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.018583 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.018761 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.018678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-host\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.024924 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.024899 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f454b995-5cf2-461d-b6f1-932bd4a45110-agent-certs\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:06.026677 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.025702 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-etc-tuned\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.026677 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.025836 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovn-node-metrics-cert\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.027923 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.027897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-tmp\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.031382 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.031344 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvlv\" (UniqueName: \"kubernetes.io/projected/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kube-api-access-dwvlv\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.032685 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.032659 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsljf\" (UniqueName: \"kubernetes.io/projected/196bf414-2c29-40a0-a957-f94cadaa7ff2-kube-api-access-vsljf\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:06.035461 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.035431 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bmk\" (UniqueName: \"kubernetes.io/projected/5ab0c10c-8503-43e7-8892-894c3c0650d4-kube-api-access-72bmk\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.035941 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.035920 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvzf2\" (UniqueName: \"kubernetes.io/projected/d9c43876-cf34-43bb-b641-01e1b6a4b143-kube-api-access-xvzf2\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:06.036930 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.036905 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9db\" (UniqueName: \"kubernetes.io/projected/8e6a1947-49a2-44d8-9db3-5004938650e6-kube-api-access-sk9db\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.037001 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.036965 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p9n\" (UniqueName: \"kubernetes.io/projected/453eddca-0701-4325-b5e2-12b4e3e78565-kube-api-access-k4p9n\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:06.037452 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.037429 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2mm\" (UniqueName: \"kubernetes.io/projected/f771de0f-1ffa-4de2-823d-40d0a95eed5c-kube-api-access-8t2mm\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.040671 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.040647 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftdc\" (UniqueName: \"kubernetes.io/projected/f07e2583-7e4e-49ef-8980-e76733289e46-kube-api-access-8ftdc\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.040769 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.040725 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjxn\" (UniqueName: \"kubernetes.io/projected/3387800d-492d-4f65-b3d8-4bbccd906d03-kube-api-access-pnjxn\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:06.121049 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.121009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:15:06.137224 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:06.137160 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196bf414_2c29_40a0_a957_f94cadaa7ff2.slice/crio-31ff3f5f369ad136854c162dcb692d3d97059b0c2109755a452b33e64cec2b9c WatchSource:0}: Error finding container 31ff3f5f369ad136854c162dcb692d3d97059b0c2109755a452b33e64cec2b9c: Status 404 returned error can't find the container with id 31ff3f5f369ad136854c162dcb692d3d97059b0c2109755a452b33e64cec2b9c Oct 31 03:15:06.146656 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.146629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:06.152711 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.152667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:06.158400 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:06.158373 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf454b995_5cf2_461d_b6f1_932bd4a45110.slice/crio-00b703925370846face39bcf6f0b1aa8b06d4c20c9cad2abdc1b7c66d8264798 WatchSource:0}: Error finding container 00b703925370846face39bcf6f0b1aa8b06d4c20c9cad2abdc1b7c66d8264798: Status 404 returned error can't find the container with id 00b703925370846face39bcf6f0b1aa8b06d4c20c9cad2abdc1b7c66d8264798 Oct 31 03:15:06.165344 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.165322 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:15:06.166009 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:06.165982 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab0c10c_8503_43e7_8892_894c3c0650d4.slice/crio-70df0484b897b8dd05644219f0263c649060426627184180e4e2ee0dfb9d2e2a WatchSource:0}: Error finding container 70df0484b897b8dd05644219f0263c649060426627184180e4e2ee0dfb9d2e2a: Status 404 returned error can't find the container with id 70df0484b897b8dd05644219f0263c649060426627184180e4e2ee0dfb9d2e2a Oct 31 03:15:06.169374 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.169188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:15:06.171847 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.171820 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:15:06.184606 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.184582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:15:06.185524 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:06.185401 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453eddca_0701_4325_b5e2_12b4e3e78565.slice/crio-9ba6f3486d62ff9e0862d8561b9a8aeb84fde0814a6ca9b5c07d4c8b84b954e3 WatchSource:0}: Error finding container 9ba6f3486d62ff9e0862d8561b9a8aeb84fde0814a6ca9b5c07d4c8b84b954e3: Status 404 returned error can't find the container with id 9ba6f3486d62ff9e0862d8561b9a8aeb84fde0814a6ca9b5c07d4c8b84b954e3 Oct 31 03:15:06.195837 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:06.195739 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf52d3e9_f738_4aaf_b8e4_de32943b6b03.slice/crio-0ab644eac17d3db659a617f20aeb7561c7269c3951589fcaa8b62c0872ba9b23 WatchSource:0}: Error finding container 0ab644eac17d3db659a617f20aeb7561c7269c3951589fcaa8b62c0872ba9b23: Status 404 returned error can't find the container with id 0ab644eac17d3db659a617f20aeb7561c7269c3951589fcaa8b62c0872ba9b23 Oct 31 03:15:06.195837 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.195753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gqnj2" Oct 31 03:15:06.204056 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.204030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:15:06.205495 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:06.205470 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6a1947_49a2_44d8_9db3_5004938650e6.slice/crio-08a8657a575ac593b1c963f6d728cec6c1f5337de88a5de0e0190df490623da7 WatchSource:0}: Error finding container 08a8657a575ac593b1c963f6d728cec6c1f5337de88a5de0e0190df490623da7: Status 404 returned error can't find the container with id 08a8657a575ac593b1c963f6d728cec6c1f5337de88a5de0e0190df490623da7 Oct 31 03:15:06.216274 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:06.216248 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3387800d_492d_4f65_b3d8_4bbccd906d03.slice/crio-6fbf68753c4f03805823cd86fc77d22f11652fafe58de6caebe1d3bb2723dcd0 WatchSource:0}: Error finding container 6fbf68753c4f03805823cd86fc77d22f11652fafe58de6caebe1d3bb2723dcd0: Status 404 returned error can't find the container with id 6fbf68753c4f03805823cd86fc77d22f11652fafe58de6caebe1d3bb2723dcd0 Oct 31 03:15:06.518360 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.518261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:06.518360 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.518322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:06.518360 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.518344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:06.518640 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.518491 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:06.518640 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.518514 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:06.518640 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.518527 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kfm56 for pod openshift-network-diagnostics/network-check-target-dlvjw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:06.518640 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.518590 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56 podName:6b1ec6c2-6309-4e90-ae0b-b85aefc8c637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:07.518570311 +0000 UTC m=+4.036934524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kfm56" (UniqueName: "kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56") pod "network-check-target-dlvjw" (UID: "6b1ec6c2-6309-4e90-ae0b-b85aefc8c637") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:06.519001 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.518984 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:06.519087 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.519031 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret podName:73113784-937f-492c-9ac1-1a98c59d43e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:07.519020029 +0000 UTC m=+4.037384234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret") pod "global-pull-secret-syncer-9cczw" (UID: "73113784-937f-492c-9ac1-1a98c59d43e6") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:06.519147 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.519090 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:06.519147 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.519126 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs podName:d9c43876-cf34-43bb-b641-01e1b6a4b143 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:07.519114097 +0000 UTC m=+4.037478311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs") pod "network-metrics-daemon-flc2b" (UID: "d9c43876-cf34-43bb-b641-01e1b6a4b143") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:06.854058 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.853887 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:10:06 +0000 UTC" deadline="2028-07-16 16:40:31.73142831 +0000 UTC" Oct 31 03:15:06.854058 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.853951 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14989h25m24.877481224s" Oct 31 03:15:06.950623 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.950584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:06.950791 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:06.950773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:06.976092 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.976052 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbm7r" event={"ID":"3387800d-492d-4f65-b3d8-4bbccd906d03","Type":"ContainerStarted","Data":"6fbf68753c4f03805823cd86fc77d22f11652fafe58de6caebe1d3bb2723dcd0"} Oct 31 03:15:06.981851 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.981813 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqnj2" event={"ID":"8e6a1947-49a2-44d8-9db3-5004938650e6","Type":"ContainerStarted","Data":"08a8657a575ac593b1c963f6d728cec6c1f5337de88a5de0e0190df490623da7"} Oct 31 03:15:06.990448 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:06.990414 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xwkgw" event={"ID":"196bf414-2c29-40a0-a957-f94cadaa7ff2","Type":"ContainerStarted","Data":"31ff3f5f369ad136854c162dcb692d3d97059b0c2109755a452b33e64cec2b9c"} Oct 31 03:15:07.002473 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.002387 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" event={"ID":"df52d3e9-f738-4aaf-b8e4-de32943b6b03","Type":"ContainerStarted","Data":"0ab644eac17d3db659a617f20aeb7561c7269c3951589fcaa8b62c0872ba9b23"} Oct 31 03:15:07.006730 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.006648 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghj2p" event={"ID":"453eddca-0701-4325-b5e2-12b4e3e78565","Type":"ContainerStarted","Data":"9ba6f3486d62ff9e0862d8561b9a8aeb84fde0814a6ca9b5c07d4c8b84b954e3"} Oct 31 03:15:07.008796 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.008718 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerStarted","Data":"912adab67e1d947caa2409ae9a5582fe858c2c79853d1304ed6a0b1d234532c9"} Oct 31 03:15:07.010745 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.010717 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" event={"ID":"f07e2583-7e4e-49ef-8980-e76733289e46","Type":"ContainerStarted","Data":"67c75e6594762a6c6226eb538460eb58187f87d8c55844b8b81ca0a2cdcfa697"} Oct 31 03:15:07.012835 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.012808 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"70df0484b897b8dd05644219f0263c649060426627184180e4e2ee0dfb9d2e2a"} Oct 31 03:15:07.017960 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.017932 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t5fpc" event={"ID":"f454b995-5cf2-461d-b6f1-932bd4a45110","Type":"ContainerStarted","Data":"00b703925370846face39bcf6f0b1aa8b06d4c20c9cad2abdc1b7c66d8264798"} Oct 31 03:15:07.527562 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.527234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:07.527562 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.527546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527392 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret podName:73113784-937f-492c-9ac1-1a98c59d43e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:09.527632834 +0000 UTC m=+6.045997042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret") pod "global-pull-secret-syncer-9cczw" (UID: "73113784-937f-492c-9ac1-1a98c59d43e6") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.527578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527672 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527730 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527745 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527758 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kfm56 for pod openshift-network-diagnostics/network-check-target-dlvjw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527768 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs podName:d9c43876-cf34-43bb-b641-01e1b6a4b143 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:09.527755565 +0000 UTC m=+6.046119778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs") pod "network-metrics-daemon-flc2b" (UID: "d9c43876-cf34-43bb-b641-01e1b6a4b143") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:07.527816 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.527793 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56 podName:6b1ec6c2-6309-4e90-ae0b-b85aefc8c637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:09.527784446 +0000 UTC m=+6.046148638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kfm56" (UniqueName: "kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56") pod "network-check-target-dlvjw" (UID: "6b1ec6c2-6309-4e90-ae0b-b85aefc8c637") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:07.854991 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.854864 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:10:06 +0000 UTC" deadline="2028-05-12 02:09:04.613952227 +0000 UTC" Oct 31 03:15:07.854991 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.854911 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13414h53m56.759046758s" Oct 31 03:15:07.958061 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.957504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:07.958061 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.957661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:07.958061 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:07.957713 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:07.958061 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:07.957843 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:08.030808 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:08.030773 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"057a5e637a257f84120786f81c645c34","Type":"ContainerStarted","Data":"40e867ec31b9920e828c419c6aaf2fbd87b2330fc196d41fbcfeae3621d8876f"} Oct 31 03:15:08.035909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:08.035880 2579 generic.go:356] "Generic (PLEG): container finished" podID="b6a0fa3bb14222e5d3d01c4ed0d8f38f" containerID="97382951ed0825677e6b5466d67f742c206dfec8d048046d1559607a9423a74d" exitCode=0 Oct 31 03:15:08.036074 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:08.035922 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"b6a0fa3bb14222e5d3d01c4ed0d8f38f","Type":"ContainerDied","Data":"97382951ed0825677e6b5466d67f742c206dfec8d048046d1559607a9423a74d"} Oct 31 03:15:08.049719 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:08.048833 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" podStartSLOduration=3.048816074 podStartE2EDuration="3.048816074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:08.048200599 +0000 UTC m=+4.566564817" watchObservedRunningTime="2026-10-31 03:15:08.048816074 +0000 UTC m=+4.567180289" Oct 31 03:15:08.952900 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:08.952860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:08.953539 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:08.953006 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:09.063492 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:09.063455 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"b6a0fa3bb14222e5d3d01c4ed0d8f38f","Type":"ContainerStarted","Data":"ab304681b02447ce6b723f4cf3792013501bda867b9ace1bbae3c02103f14783"} Oct 31 03:15:09.547627 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:09.547539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:09.547627 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:09.547596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:09.547627 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:09.547627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:09.547915 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.547784 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:09.547915 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.547867 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret podName:73113784-937f-492c-9ac1-1a98c59d43e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:13.547845095 +0000 UTC m=+10.066209301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret") pod "global-pull-secret-syncer-9cczw" (UID: "73113784-937f-492c-9ac1-1a98c59d43e6") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:09.548024 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.547962 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:09.548024 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.547983 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:09.548024 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.547998 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kfm56 for pod openshift-network-diagnostics/network-check-target-dlvjw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:09.548177 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.548045 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56 podName:6b1ec6c2-6309-4e90-ae0b-b85aefc8c637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:13.548033436 +0000 UTC m=+10.066397640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kfm56" (UniqueName: "kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56") pod "network-check-target-dlvjw" (UID: "6b1ec6c2-6309-4e90-ae0b-b85aefc8c637") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:09.548177 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.548107 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:09.548177 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.548132 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs podName:d9c43876-cf34-43bb-b641-01e1b6a4b143 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:13.548124446 +0000 UTC m=+10.066488637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs") pod "network-metrics-daemon-flc2b" (UID: "d9c43876-cf34-43bb-b641-01e1b6a4b143") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:09.952675 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:09.951819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:09.952675 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.951950 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:09.952675 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:09.952388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:09.952675 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:09.952493 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:10.074332 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:10.074241 2579 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="e714158845edecc29e563084c43fd3127f86775c95f4e582d59d49723720c0c3" exitCode=0 Oct 31 03:15:10.074505 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:10.074372 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerDied","Data":"e714158845edecc29e563084c43fd3127f86775c95f4e582d59d49723720c0c3"} Oct 31 03:15:10.102503 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:10.102442 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" podStartSLOduration=5.102420429 podStartE2EDuration="5.102420429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:09.083834095 +0000 UTC m=+5.602198312" watchObservedRunningTime="2026-10-31 03:15:10.102420429 +0000 UTC m=+6.620784647" Oct 31 03:15:10.950064 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:10.949862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:10.963321 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:10.950006 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:11.079918 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:11.079874 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t5fpc" event={"ID":"f454b995-5cf2-461d-b6f1-932bd4a45110","Type":"ContainerStarted","Data":"93910fb60eb602a77bbc6ba501480f5c8825b00ba3a0281119db92f6b817fc6d"} Oct 31 03:15:11.951874 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:11.951836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:11.952047 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:11.951980 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:11.952124 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:11.952076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:11.952183 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:11.952164 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:12.089053 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:12.088980 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xwkgw" event={"ID":"196bf414-2c29-40a0-a957-f94cadaa7ff2","Type":"ContainerStarted","Data":"99ec14aee73f10092e57c2ac606432d23ec1e8438e1e8ec0aead73b2b1d11a12"} Oct 31 03:15:12.106854 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:12.105649 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xwkgw" podStartSLOduration=2.613259021 podStartE2EDuration="8.105630139s" totalImagesPullingTime="5.492371118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:12.105525444 +0000 UTC m=+8.623889661" watchObservedRunningTime="2026-10-31 03:15:12.105630139 +0000 UTC m=+8.623994356" Oct 31 03:15:12.106854 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:12.106142 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t5fpc" podStartSLOduration=3.6305359619999997 podStartE2EDuration="8.106133665s" totalImagesPullingTime="4.475597703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:11.101255967 +0000 UTC m=+7.619620186" watchObservedRunningTime="2026-10-31 03:15:12.106133665 +0000 UTC m=+8.624497880" Oct 31 03:15:12.950643 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:12.950112 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:12.950643 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:12.950275 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:13.093899 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:13.093828 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" event={"ID":"df52d3e9-f738-4aaf-b8e4-de32943b6b03","Type":"ContainerStarted","Data":"3729a3433ca16edbfd07e1832c80a2f1d3aec57c0bd83218c5eee3aae639d8e7"} Oct 31 03:15:13.592068 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:13.591817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:13.592068 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:13.591863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:13.592068 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:13.591888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:13.592068 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592061 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:13.592068 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592078 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:13.592518 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592091 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kfm56 for pod openshift-network-diagnostics/network-check-target-dlvjw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:13.592518 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592151 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56 podName:6b1ec6c2-6309-4e90-ae0b-b85aefc8c637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.592131637 +0000 UTC m=+18.110495836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kfm56" (UniqueName: "kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56") pod "network-check-target-dlvjw" (UID: "6b1ec6c2-6309-4e90-ae0b-b85aefc8c637") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:13.592628 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592581 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:13.592739 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592630 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret podName:73113784-937f-492c-9ac1-1a98c59d43e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.592615386 +0000 UTC m=+18.110979581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret") pod "global-pull-secret-syncer-9cczw" (UID: "73113784-937f-492c-9ac1-1a98c59d43e6") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:13.592739 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592689 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:13.592739 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.592719 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs podName:d9c43876-cf34-43bb-b641-01e1b6a4b143 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.592709028 +0000 UTC m=+18.111073216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs") pod "network-metrics-daemon-flc2b" (UID: "d9c43876-cf34-43bb-b641-01e1b6a4b143") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:13.951647 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:13.951617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:13.951854 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.951748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:13.952152 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:13.952132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:13.952276 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:13.952253 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:14.093593 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:14.093553 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:14.094950 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:14.094931 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:14.302338 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:14.302162 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:15:14.864267 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:14.863408 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:15:14.302176582Z","UUID":"21930243-a571-4bcb-b419-9e2887205272","Handler":null,"Name":"","Endpoint":""} Oct 31 03:15:14.865370 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:14.865348 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:15:14.865370 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:14.865377 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:15:14.949775 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:14.949748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:14.949933 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:14.949900 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:15.110250 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.110154 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerStarted","Data":"8b11705232151f4fc803cd9bf5c432dd4fa1ffea5fa6f0225eeea84788bd0a94"} Oct 31 03:15:15.113360 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.113333 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbm7r" event={"ID":"3387800d-492d-4f65-b3d8-4bbccd906d03","Type":"ContainerStarted","Data":"b7067275c087b43b2f3b0a661e1a3feb9598f86a41cd3bd55ea7063666fdf490"} Oct 31 03:15:15.115606 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.115581 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gqnj2" event={"ID":"8e6a1947-49a2-44d8-9db3-5004938650e6","Type":"ContainerStarted","Data":"b33a8fe445753109ea4adebb251c46c0cc4abeb836a4a973f989c6a913e760f1"} Oct 31 03:15:15.117903 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.117876 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" event={"ID":"df52d3e9-f738-4aaf-b8e4-de32943b6b03","Type":"ContainerStarted","Data":"dce9897adb96b71cc0a4692261d493a44a3255978052d377cfc58c93dcf06df6"} Oct 31 03:15:15.149994 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.149414 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gqnj2" podStartSLOduration=2.74778876 podStartE2EDuration="11.149394544s" totalImagesPullingTime="8.401605784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:15.148511529 +0000 UTC m=+11.666875741" watchObservedRunningTime="2026-10-31 03:15:15.149394544 +0000 UTC m=+11.667758755" Oct 31 03:15:15.163476 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.163421 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dbm7r" podStartSLOduration=2.867271063 podStartE2EDuration="11.163403654s" totalImagesPullingTime="8.296132591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:15.163360207 +0000 UTC m=+11.681724421" watchObservedRunningTime="2026-10-31 03:15:15.163403654 +0000 UTC m=+11.681767867" Oct 31 03:15:15.949711 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.949462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:15.949862 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:15.949558 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:15.949862 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:15.949825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:15.950017 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:15.949995 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:16.123052 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:16.122964 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" event={"ID":"df52d3e9-f738-4aaf-b8e4-de32943b6b03","Type":"ContainerStarted","Data":"262fbed85580ce5a127bf11b8045b268ae9a32746ddf40520383bf41d1225400"} Oct 31 03:15:16.125291 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:16.125259 2579 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="8b11705232151f4fc803cd9bf5c432dd4fa1ffea5fa6f0225eeea84788bd0a94" exitCode=0 Oct 31 03:15:16.125400 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:16.125347 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerDied","Data":"8b11705232151f4fc803cd9bf5c432dd4fa1ffea5fa6f0225eeea84788bd0a94"} Oct 31 03:15:16.145792 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:16.145734 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" podStartSLOduration=3.288959318 podStartE2EDuration="12.14572054s" totalImagesPullingTime="8.856761222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:16.144894026 +0000 UTC m=+12.663258239" watchObservedRunningTime="2026-10-31 03:15:16.14572054 +0000 UTC m=+12.664084752" Oct 31 03:15:16.950106 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:16.950067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:16.950275 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:16.950185 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:17.128859 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:17.128817 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghj2p" event={"ID":"453eddca-0701-4325-b5e2-12b4e3e78565","Type":"ContainerStarted","Data":"394dc2683bd7f203e28da4eea13da93c0f823931289b85d37dadf3ee540d3890"} Oct 31 03:15:17.151325 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:17.151280 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ghj2p" podStartSLOduration=4.532781636 podStartE2EDuration="13.151265205s" totalImagesPullingTime="8.618483569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:17.150765041 +0000 UTC m=+13.669129253" watchObservedRunningTime="2026-10-31 03:15:17.151265205 +0000 UTC m=+13.669629416" Oct 31 03:15:17.949529 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:17.949488 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:17.949687 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:17.949623 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:17.949767 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:17.949682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:17.949817 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:17.949797 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:18.132926 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:18.132891 2579 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="48fbebf18ffb7d0d342d7923b0658462078ab4ba87dd41db81f3ae7e4d87d0dc" exitCode=0 Oct 31 03:15:18.133545 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:18.133233 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerDied","Data":"48fbebf18ffb7d0d342d7923b0658462078ab4ba87dd41db81f3ae7e4d87d0dc"} Oct 31 03:15:18.134842 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:18.134818 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" event={"ID":"f07e2583-7e4e-49ef-8980-e76733289e46","Type":"ContainerStarted","Data":"6c26626cc2aefded592566152e99528f05b9f3be7a6c2b5fd5c2de772c57a876"} Oct 31 03:15:18.180233 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:18.180158 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" podStartSLOduration=3.294320704 podStartE2EDuration="14.180141612s" totalImagesPullingTime="10.885820908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:18.179818643 +0000 UTC m=+14.698182857" watchObservedRunningTime="2026-10-31 03:15:18.180141612 +0000 UTC m=+14.698505824" Oct 31 03:15:18.949276 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:18.949246 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:18.949463 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:18.949371 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:19.949911 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:19.949874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:19.950624 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:19.950053 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:19.950624 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:19.950108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:19.950624 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:19.950237 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:20.142039 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:20.141999 2579 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="810f4686f7984e446e9472ea40d1ecbd573c56c6a71980da65675f8d5b621e57" exitCode=0 Oct 31 03:15:20.142187 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:20.142060 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerDied","Data":"810f4686f7984e446e9472ea40d1ecbd573c56c6a71980da65675f8d5b621e57"} Oct 31 03:15:20.949263 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:20.949225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:20.949453 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:20.949344 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:21.654971 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:21.654747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:21.654994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:21.655021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.654931 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.655114 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret podName:73113784-937f-492c-9ac1-1a98c59d43e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:37.655093446 +0000 UTC m=+34.173457642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret") pod "global-pull-secret-syncer-9cczw" (UID: "73113784-937f-492c-9ac1-1a98c59d43e6") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.655148 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.655166 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.655189 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.655218 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kfm56 for pod openshift-network-diagnostics/network-check-target-dlvjw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.655231 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs podName:d9c43876-cf34-43bb-b641-01e1b6a4b143 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:37.655213259 +0000 UTC m=+34.173577462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs") pod "network-metrics-daemon-flc2b" (UID: "d9c43876-cf34-43bb-b641-01e1b6a4b143") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:21.655471 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.655253 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56 podName:6b1ec6c2-6309-4e90-ae0b-b85aefc8c637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:37.655243455 +0000 UTC m=+34.173607644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kfm56" (UniqueName: "kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56") pod "network-check-target-dlvjw" (UID: "6b1ec6c2-6309-4e90-ae0b-b85aefc8c637") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:21.949814 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:21.949627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:21.949976 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:21.949663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:21.949976 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.949930 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:21.949976 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:21.949951 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:22.148305 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.148226 2579 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="0282c04de41c04e57c8eb4d4d09bb07f39bd2464d3b89bcbe83ea4ade0497aee" exitCode=0 Oct 31 03:15:22.148305 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.148300 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerDied","Data":"0282c04de41c04e57c8eb4d4d09bb07f39bd2464d3b89bcbe83ea4ade0497aee"} Oct 31 03:15:22.151274 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.151251 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"4629436c2a3dd3616f78d766402b0d6cef5a6882393049927543417296aa9042"} Oct 31 03:15:22.151389 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.151320 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"acaf32fb60bc6b863881704917a1bd38ecc7c12000dfc932086087deab48f5e5"} Oct 31 03:15:22.151389 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.151351 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"257fb24b904ef9c42f839215f1caa04963c5997a353e95c0c2993175bb4a75a7"} Oct 31 03:15:22.151389 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.151387 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"26acfba8e7cf95b142e59b22765f453c73521ac4ec7cabd81517c3687e86c4ed"} Oct 31 03:15:22.151565 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.151413 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"ed34096c01fb92ad937a83085a4e16e974ca46c4fcf8d23579214bd2316988c3"} Oct 31 03:15:22.151565 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.151439 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"a564a79b6af27e56785a315ad15ca45dce21eb720aff967b28ab4d5f7f692ba4"} Oct 31 03:15:22.950102 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:22.950061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:22.950535 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:22.950221 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:23.156598 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:23.156565 2579 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="70c2bff2c716a2d9b94078b1ad43feefba802bdcbdd760cdd95978d05d001a3e" exitCode=0 Oct 31 03:15:23.156738 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:23.156621 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerDied","Data":"70c2bff2c716a2d9b94078b1ad43feefba802bdcbdd760cdd95978d05d001a3e"} Oct 31 03:15:23.755645 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:23.755615 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbm7r_3387800d-492d-4f65-b3d8-4bbccd906d03/dns-node-resolver/0.log" Oct 31 03:15:23.950572 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:23.950531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:23.951152 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:23.950614 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:23.951152 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:23.950657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:23.951152 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:23.950704 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:24.161533 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:24.161491 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" event={"ID":"f771de0f-1ffa-4de2-823d-40d0a95eed5c","Type":"ContainerStarted","Data":"944ffdef2e0f86275d9cf832bc103058746bbccf9291580688fdfda2746a4e58"} Oct 31 03:15:24.164390 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:24.164369 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"78932fdcc6b937866c04d56047f8b378ff37882d8ae4f411c9571ca59b75d09f"} Oct 31 03:15:24.190839 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:24.190794 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rrqw6" podStartSLOduration=8.361822332 podStartE2EDuration="20.19078171s" totalImagesPullingTime="11.828959378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:24.18884324 +0000 UTC m=+20.707207450" watchObservedRunningTime="2026-10-31 03:15:24.19078171 +0000 UTC m=+20.709145958" Oct 31 03:15:24.950108 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:24.950078 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:24.950268 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:24.950179 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:25.130680 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:25.130653 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xwkgw_196bf414-2c29-40a0-a957-f94cadaa7ff2/node-ca/0.log" Oct 31 03:15:25.949281 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:25.949252 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:25.949439 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:25.949341 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:25.949439 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:25.949360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:25.949521 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:25.949485 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:26.172051 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.172022 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" event={"ID":"5ab0c10c-8503-43e7-8892-894c3c0650d4","Type":"ContainerStarted","Data":"ec029c48e4bb18ad1a15114b33bb32b522ac616bded0ce908e3c11d4d772474b"} Oct 31 03:15:26.172733 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.172287 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:26.172733 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.172368 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:26.187714 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.187694 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:26.205186 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.205127 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" podStartSLOduration=7.271118905 podStartE2EDuration="22.205116857s" totalImagesPullingTime="14.933997952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:26.203229871 +0000 UTC m=+22.721594083" watchObservedRunningTime="2026-10-31 03:15:26.205116857 +0000 UTC m=+22.723481067" Oct 31 03:15:26.949897 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.949856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:26.950119 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:26.949970 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:26.964583 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.964555 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:26.964696 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.964654 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:15:26.965055 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:26.965039 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:15:27.174902 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:27.174871 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:27.190103 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:27.190081 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:27.949429 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:27.949398 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:27.949595 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:27.949413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:27.949595 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:27.949574 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:27.949729 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:27.949684 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:28.949632 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:28.949598 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:28.949997 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:28.949705 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:29.949944 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:29.949893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:29.949944 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:29.949936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:29.950417 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:29.950040 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:29.950417 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:29.950149 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:30.949319 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:30.949284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:30.949563 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:30.949397 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:31.949707 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:31.949674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:31.950116 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:31.949799 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:31.950116 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:31.949856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:31.950116 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:31.949954 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:32.949753 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:32.949509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:32.950241 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:32.949961 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:33.950392 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:33.950350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:33.950848 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:33.950498 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:33.950848 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:33.950555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:33.950848 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:33.950646 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:34.949490 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:34.949452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:34.949688 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:34.949584 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:35.949587 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:35.949549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:35.950097 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:35.949549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:35.950097 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:35.949685 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:35.950097 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:35.949732 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:36.949508 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:36.949473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:36.949683 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:36.949582 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:37.668852 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:37.668813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:37.669044 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:37.668860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:37.669044 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:37.668884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:37.669044 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.668980 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:37.669044 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.669033 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:37.669281 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.669050 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret podName:73113784-937f-492c-9ac1-1a98c59d43e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:16:09.669035548 +0000 UTC m=+66.187399753 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret") pod "global-pull-secret-syncer-9cczw" (UID: "73113784-937f-492c-9ac1-1a98c59d43e6") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:37.669281 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.669054 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:37.669281 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.668980 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:37.669281 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.669068 2579 projected.go:196] Error preparing data for projected volume kube-api-access-kfm56 for pod openshift-network-diagnostics/network-check-target-dlvjw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:37.669281 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.669096 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs podName:d9c43876-cf34-43bb-b641-01e1b6a4b143 nodeName:}" failed. No retries permitted until 2026-10-31 03:16:09.669085427 +0000 UTC m=+66.187449617 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs") pod "network-metrics-daemon-flc2b" (UID: "d9c43876-cf34-43bb-b641-01e1b6a4b143") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:37.669281 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.669117 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56 podName:6b1ec6c2-6309-4e90-ae0b-b85aefc8c637 nodeName:}" failed. No retries permitted until 2026-10-31 03:16:09.669102918 +0000 UTC m=+66.187467133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kfm56" (UniqueName: "kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56") pod "network-check-target-dlvjw" (UID: "6b1ec6c2-6309-4e90-ae0b-b85aefc8c637") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:37.949269 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:37.949235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:37.949269 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:37.949270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:37.949513 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.949338 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:37.949513 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:37.949449 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:38.949997 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:38.949966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:38.950417 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:38.950077 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:39.949665 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:39.949630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:39.949837 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:39.949627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:39.949837 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:39.949738 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:39.949837 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:39.949812 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:40.950133 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:40.950084 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:40.950665 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:40.950262 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:41.437666 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:41.437634 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9cczw"] Oct 31 03:15:41.437881 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:41.437775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:41.437948 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:41.437890 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:41.440913 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:41.440881 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dlvjw"] Oct 31 03:15:41.441059 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:41.441005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:41.441158 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:41.441096 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:41.441550 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:41.441527 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flc2b"] Oct 31 03:15:41.441768 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:41.441744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:41.441870 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:41.441854 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:42.950217 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:42.950178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:42.950594 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:42.950248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:42.950594 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:42.950307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:42.950594 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:42.950248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:42.950594 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:42.950361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:42.950594 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:42.950421 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:44.949842 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:44.949808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:44.950277 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:44.949870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:44.950277 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:44.949896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:44.950277 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:44.949981 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9cczw" podUID="73113784-937f-492c-9ac1-1a98c59d43e6" Oct 31 03:15:44.950277 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:44.950025 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flc2b" podUID="d9c43876-cf34-43bb-b641-01e1b6a4b143" Oct 31 03:15:44.950277 ip-10-0-141-153 kubenswrapper[2579]: E1031 03:15:44.950094 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dlvjw" podUID="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" Oct 31 03:15:46.248135 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.248063 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeReady" Oct 31 03:15:46.248500 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.248189 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 03:15:46.302508 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.302468 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vbv64"] Oct 31 03:15:46.314113 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.314090 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s8l89"] Oct 31 03:15:46.314242 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.314225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.318376 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.318140 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:15:46.318376 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.318311 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:15:46.318555 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.318335 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zf8rk\"" Oct 31 03:15:46.318555 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.318362 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:15:46.333788 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.333759 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbv64"] Oct 31 03:15:46.333788 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.333795 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s8l89"] Oct 31 03:15:46.334016 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.333895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.399712 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.399681 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-76rrr"] Oct 31 03:15:46.413911 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.413875 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-76rrr"] Oct 31 03:15:46.414058 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.413983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.418603 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418578 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:15:46.418741 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418614 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:15:46.418741 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418683 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:15:46.418741 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418730 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dk5h8\"" Oct 31 03:15:46.418741 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418731 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:15:46.418944 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418739 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dct2q\"" Oct 31 03:15:46.418944 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418760 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:15:46.418944 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.418764 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:15:46.421163 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.421143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f09f77c2-e722-4f39-b4c9-2fcca44268a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.421286 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.421232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f09f77c2-e722-4f39-b4c9-2fcca44268a9-crio-socket\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.421354 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.421325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f09f77c2-e722-4f39-b4c9-2fcca44268a9-data-volume\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.421408 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.421364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8z5\" (UniqueName: \"kubernetes.io/projected/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-api-access-kc8z5\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.421456 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.421402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.421501 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.421468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrs6\" (UniqueName: \"kubernetes.io/projected/4080580a-a99a-49e0-a255-06cb08581290-kube-api-access-jzrs6\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.421535 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.421511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4080580a-a99a-49e0-a255-06cb08581290-cert\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.522291 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5deb6a1-16c9-4491-8e07-d8deafd8607f-config-volume\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.522291 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.522291 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrs6\" (UniqueName: \"kubernetes.io/projected/4080580a-a99a-49e0-a255-06cb08581290-kube-api-access-jzrs6\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.522542 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4080580a-a99a-49e0-a255-06cb08581290-cert\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.522542 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5deb6a1-16c9-4491-8e07-d8deafd8607f-metrics-tls\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.522542 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f09f77c2-e722-4f39-b4c9-2fcca44268a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.522542 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5deb6a1-16c9-4491-8e07-d8deafd8607f-tmp-dir\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.522542 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgtd\" (UniqueName: \"kubernetes.io/projected/e5deb6a1-16c9-4491-8e07-d8deafd8607f-kube-api-access-hbgtd\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.522726 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f09f77c2-e722-4f39-b4c9-2fcca44268a9-crio-socket\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.522726 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f09f77c2-e722-4f39-b4c9-2fcca44268a9-data-volume\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.522726 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8z5\" (UniqueName: \"kubernetes.io/projected/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-api-access-kc8z5\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.522864 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f09f77c2-e722-4f39-b4c9-2fcca44268a9-crio-socket\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.522999 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.522976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f09f77c2-e722-4f39-b4c9-2fcca44268a9-data-volume\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.523168 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.523151 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.526505 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.526485 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f09f77c2-e722-4f39-b4c9-2fcca44268a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.526626 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.526524 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4080580a-a99a-49e0-a255-06cb08581290-cert\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.531040 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.531008 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8z5\" (UniqueName: \"kubernetes.io/projected/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-api-access-kc8z5\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.531404 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.531383 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrs6\" (UniqueName: \"kubernetes.io/projected/4080580a-a99a-49e0-a255-06cb08581290-kube-api-access-jzrs6\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.623211 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.623174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5deb6a1-16c9-4491-8e07-d8deafd8607f-config-volume\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.623364 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.623232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5deb6a1-16c9-4491-8e07-d8deafd8607f-metrics-tls\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.623364 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.623253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5deb6a1-16c9-4491-8e07-d8deafd8607f-tmp-dir\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.623364 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.623286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgtd\" (UniqueName: \"kubernetes.io/projected/e5deb6a1-16c9-4491-8e07-d8deafd8607f-kube-api-access-hbgtd\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.623684 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.623665 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5deb6a1-16c9-4491-8e07-d8deafd8607f-tmp-dir\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.623754 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.623735 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5deb6a1-16c9-4491-8e07-d8deafd8607f-config-volume\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.625659 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.625639 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5deb6a1-16c9-4491-8e07-d8deafd8607f-metrics-tls\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.632728 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.632710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgtd\" (UniqueName: \"kubernetes.io/projected/e5deb6a1-16c9-4491-8e07-d8deafd8607f-kube-api-access-hbgtd\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.644799 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.644776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:15:46.698514 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.698487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:15:46.750946 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.750919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:46.949840 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.949809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:15:46.949840 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.949833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:15:46.950220 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.949822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:15:46.954522 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.954468 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:15:46.954522 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.954494 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:15:46.954727 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.954627 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\"" Oct 31 03:15:46.955988 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.955967 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\"" Oct 31 03:15:46.956107 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.955988 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:15:46.956107 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.956064 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:15:46.956626 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.956108 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\"" Oct 31 03:15:46.969966 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.969943 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vbv64"] Oct 31 03:15:46.986423 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:46.986058 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09f77c2_e722_4f39_b4c9_2fcca44268a9.slice/crio-029aaed000d33004247b040142c85c5550d1e3235981d0cc3b3a16ab9d938c40 WatchSource:0}: Error finding container 029aaed000d33004247b040142c85c5550d1e3235981d0cc3b3a16ab9d938c40: Status 404 returned error can't find the container with id 029aaed000d33004247b040142c85c5550d1e3235981d0cc3b3a16ab9d938c40 Oct 31 03:15:46.986423 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:46.986310 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s8l89"] Oct 31 03:15:47.033098 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:47.033073 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-76rrr"] Oct 31 03:15:47.036611 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:47.036575 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5deb6a1_16c9_4491_8e07_d8deafd8607f.slice/crio-4a2cbbba7fc88a2fa1c4789653fdaa6da169cfe897e952b3fb77bd103cf08455 WatchSource:0}: Error finding container 4a2cbbba7fc88a2fa1c4789653fdaa6da169cfe897e952b3fb77bd103cf08455: Status 404 returned error can't find the container with id 4a2cbbba7fc88a2fa1c4789653fdaa6da169cfe897e952b3fb77bd103cf08455 Oct 31 03:15:47.219284 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:47.219034 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbv64" event={"ID":"4080580a-a99a-49e0-a255-06cb08581290","Type":"ContainerStarted","Data":"31520d8049803b599e914cd7b0a04ac635442220a46039e330942f90b50efcaa"} Oct 31 03:15:47.220220 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:47.220172 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76rrr" event={"ID":"e5deb6a1-16c9-4491-8e07-d8deafd8607f","Type":"ContainerStarted","Data":"4a2cbbba7fc88a2fa1c4789653fdaa6da169cfe897e952b3fb77bd103cf08455"} Oct 31 03:15:47.221553 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:47.221535 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s8l89" event={"ID":"f09f77c2-e722-4f39-b4c9-2fcca44268a9","Type":"ContainerStarted","Data":"b4ab35c40e7a4759b11cce93406e6fb8802fa3cb015ef38da4baebd9731abfe6"} Oct 31 03:15:47.221632 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:47.221575 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s8l89" event={"ID":"f09f77c2-e722-4f39-b4c9-2fcca44268a9","Type":"ContainerStarted","Data":"029aaed000d33004247b040142c85c5550d1e3235981d0cc3b3a16ab9d938c40"} Oct 31 03:15:48.226818 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.226776 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s8l89" event={"ID":"f09f77c2-e722-4f39-b4c9-2fcca44268a9","Type":"ContainerStarted","Data":"d0bf61ee0766a942f7d3da087b34ee793aa933f0e410b786297744c89e083813"} Oct 31 03:15:48.742431 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.742251 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz"] Oct 31 03:15:48.748426 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.748402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.765713 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.765690 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 03:15:48.769405 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.769353 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 03:15:48.769900 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.769880 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ztddf\"" Oct 31 03:15:48.777730 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.777710 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 03:15:48.787962 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.786589 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:15:48.787962 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.786840 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:15:48.788483 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.788467 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66fc4df878-chnw2"] Oct 31 03:15:48.804946 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.804924 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz"] Oct 31 03:15:48.805069 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.804976 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j"] Oct 31 03:15:48.805142 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.805089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.816702 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.815969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:48.828401 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.828370 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j"] Oct 31 03:15:48.838461 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.838431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.838551 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.838503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.838619 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.838580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f26e8e-228a-4b68-8851-c580c827e242-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.838676 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.838650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtz9\" (UniqueName: \"kubernetes.io/projected/75f26e8e-228a-4b68-8851-c580c827e242-kube-api-access-srtz9\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.852122 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.852092 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fc4df878-chnw2"] Oct 31 03:15:48.859187 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.859164 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 03:15:48.859187 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.859167 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 03:15:48.859376 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.859175 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 03:15:48.859376 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.859261 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 03:15:48.859471 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.859396 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 03:15:48.859471 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.859416 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 03:15:48.868854 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.868833 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 03:15:48.868967 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.868922 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 03:15:48.869124 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.869109 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 03:15:48.869478 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.869459 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9h5x8\"" Oct 31 03:15:48.869764 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.869747 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nz4s9\"" Oct 31 03:15:48.869871 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.869759 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 03:15:48.874380 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.874363 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gsvct"] Oct 31 03:15:48.890560 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.890532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:48.916454 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.916432 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 03:15:48.939290 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.939451 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.939451 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.939451 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0577b36e-dd61-41bb-899d-c7af7b388696-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:48.939451 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:48.939451 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngg2k\" (UniqueName: \"kubernetes.io/projected/0577b36e-dd61-41bb-899d-c7af7b388696-kube-api-access-ngg2k\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:48.939699 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.939699 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f26e8e-228a-4b68-8851-c580c827e242-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.939699 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtz9\" (UniqueName: \"kubernetes.io/projected/75f26e8e-228a-4b68-8851-c580c827e242-kube-api-access-srtz9\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.939699 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.939699 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vnm\" (UniqueName: \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.939699 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:48.939984 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.939984 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:48.939984 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:48.939984 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.939984 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.939820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:48.941818 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.941792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f26e8e-228a-4b68-8851-c580c827e242-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.944563 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.944518 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.944678 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.944563 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.962086 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.962056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtz9\" (UniqueName: \"kubernetes.io/projected/75f26e8e-228a-4b68-8851-c580c827e242-kube-api-access-srtz9\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:48.968619 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.968587 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p2pq6\"" Oct 31 03:15:48.968869 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.968814 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 03:15:48.968869 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:48.968845 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 03:15:49.041030 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.040836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vnm\" (UniqueName: \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.041030 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.040886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-metrics-client-ca\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041030 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.040917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041030 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.040949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.041327 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.041327 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.041327 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.041418 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.041418 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-tls\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041489 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.041489 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.041489 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-textfile\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041489 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-sys\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041635 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.041635 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0577b36e-dd61-41bb-899d-c7af7b388696-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.041635 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.041635 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-root\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041635 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngg2k\" (UniqueName: \"kubernetes.io/projected/0577b36e-dd61-41bb-899d-c7af7b388696-kube-api-access-ngg2k\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.041787 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-wtmp\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041787 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.041787 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041681 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.042156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.042141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.042156 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.042151 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.042230 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.041685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.042230 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.042188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/85ec7f68-370c-4e14-a990-877b350fcdf8-kube-api-access-ffd55\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.042467 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.042445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.042897 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.042880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.043489 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.043452 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.043742 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.043717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0577b36e-dd61-41bb-899d-c7af7b388696-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.046708 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.046682 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.047271 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.047245 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.047520 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.047474 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.048078 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.048054 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.054581 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.054558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vnm\" (UniqueName: \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.054909 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.054889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngg2k\" (UniqueName: \"kubernetes.io/projected/0577b36e-dd61-41bb-899d-c7af7b388696-kube-api-access-ngg2k\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.079282 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.079092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:15:49.128238 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.126653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-wtmp\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/85ec7f68-370c-4e14-a990-877b350fcdf8-kube-api-access-ffd55\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-wtmp\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-metrics-client-ca\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-tls\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-textfile\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-sys\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-root\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.143573 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143465 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-root\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.144132 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143742 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.144132 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143801 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-metrics-client-ca\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.144132 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.143972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-sys\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.144553 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.144386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-textfile\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.146527 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.146506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-tls\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.147105 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.147086 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.156577 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.155005 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/85ec7f68-370c-4e14-a990-877b350fcdf8-kube-api-access-ffd55\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.156577 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.155130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:15:49.219855 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.219812 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:15:49.237726 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:49.237694 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ec7f68_370c_4e14_a990_877b350fcdf8.slice/crio-15ca714cb6a6596bc767cc6f24cd305afc64c1626a52c5fc9843c5894fb72073 WatchSource:0}: Error finding container 15ca714cb6a6596bc767cc6f24cd305afc64c1626a52c5fc9843c5894fb72073: Status 404 returned error can't find the container with id 15ca714cb6a6596bc767cc6f24cd305afc64c1626a52c5fc9843c5894fb72073 Oct 31 03:15:49.383134 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.382971 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz"] Oct 31 03:15:49.407793 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:49.407763 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f26e8e_228a_4b68_8851_c580c827e242.slice/crio-eebb12123ada46fdc68dc01673e2280e7e521dfe1686a72d03e382c9f231d67a WatchSource:0}: Error finding container eebb12123ada46fdc68dc01673e2280e7e521dfe1686a72d03e382c9f231d67a: Status 404 returned error can't find the container with id eebb12123ada46fdc68dc01673e2280e7e521dfe1686a72d03e382c9f231d67a Oct 31 03:15:49.418921 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.418891 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66fc4df878-chnw2"] Oct 31 03:15:49.423035 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:49.423003 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbe170d_bfcb_4e38_8182_59fdcb8b1628.slice/crio-fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed WatchSource:0}: Error finding container fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed: Status 404 returned error can't find the container with id fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed Oct 31 03:15:49.464725 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.464700 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j"] Oct 31 03:15:49.468422 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:49.468395 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0577b36e_dd61_41bb_899d_c7af7b388696.slice/crio-7238234ccd5cb864ec2568d80406c485b9203a4c483e6a3af0dcf0265dcb8337 WatchSource:0}: Error finding container 7238234ccd5cb864ec2568d80406c485b9203a4c483e6a3af0dcf0265dcb8337: Status 404 returned error can't find the container with id 7238234ccd5cb864ec2568d80406c485b9203a4c483e6a3af0dcf0265dcb8337 Oct 31 03:15:49.755786 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.755565 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:15:49.775180 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.775148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.777069 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.776964 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.780231 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.781708 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.781859 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.781928 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.781990 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.782073 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.782137 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.782179 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7h7nd\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.782233 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 03:15:49.782612 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.782310 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 03:15:49.951546 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951546 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951783 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951783 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951783 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951919 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951919 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951919 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.951919 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.952075 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.951927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkfz\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.952075 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.952014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.952163 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.952109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:49.952229 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:49.952177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053376 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053376 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053620 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053892 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053892 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkfz\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053892 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053892 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.053892 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.053796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.055547 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.055518 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.073221 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.073166 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.073369 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.073318 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.073442 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.073363 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.073442 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.073433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.073765 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.073743 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.073864 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.073842 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.074354 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.074330 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.074807 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.074780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.074884 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.074758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.079771 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.079355 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkfz\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.084849 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.084823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.085010 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.084990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.142238 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.142107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:15:50.239139 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.238179 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76rrr" event={"ID":"e5deb6a1-16c9-4491-8e07-d8deafd8607f","Type":"ContainerStarted","Data":"53e84184b4f34a5908affa24472acca2d9fe32724f8a22690b2f0ae6adc655c2"} Oct 31 03:15:50.239139 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.238300 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76rrr" event={"ID":"e5deb6a1-16c9-4491-8e07-d8deafd8607f","Type":"ContainerStarted","Data":"63b0b03b1cc0ae5f3466777a6e9cfa319cc71f46438a5100ed8da03b14d57447"} Oct 31 03:15:50.239139 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.239099 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-76rrr" Oct 31 03:15:50.245053 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.244975 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" event={"ID":"0577b36e-dd61-41bb-899d-c7af7b388696","Type":"ContainerStarted","Data":"7238234ccd5cb864ec2568d80406c485b9203a4c483e6a3af0dcf0265dcb8337"} Oct 31 03:15:50.247245 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.247144 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc4df878-chnw2" event={"ID":"0fbe170d-bfcb-4e38-8182-59fdcb8b1628","Type":"ContainerStarted","Data":"fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed"} Oct 31 03:15:50.250216 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.250113 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" event={"ID":"75f26e8e-228a-4b68-8851-c580c827e242","Type":"ContainerStarted","Data":"f4116b8baa8b5f42d5f239db208cbcde89a6ce55c4e9e5b9243974b90a8dcd63"} Oct 31 03:15:50.250216 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.250150 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" event={"ID":"75f26e8e-228a-4b68-8851-c580c827e242","Type":"ContainerStarted","Data":"b6f09419bed636c5714e6ece6ca085de735f3a9b4a4499c84131e647cbbb946e"} Oct 31 03:15:50.250216 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.250169 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" event={"ID":"75f26e8e-228a-4b68-8851-c580c827e242","Type":"ContainerStarted","Data":"eebb12123ada46fdc68dc01673e2280e7e521dfe1686a72d03e382c9f231d67a"} Oct 31 03:15:50.251592 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.251573 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsvct" event={"ID":"85ec7f68-370c-4e14-a990-877b350fcdf8","Type":"ContainerStarted","Data":"15ca714cb6a6596bc767cc6f24cd305afc64c1626a52c5fc9843c5894fb72073"} Oct 31 03:15:50.465972 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.465691 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-76rrr" podStartSLOduration=1.9005184640000001 podStartE2EDuration="4.465670142s" totalImagesPullingTime="2.565151678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:50.259717786 +0000 UTC m=+46.778082005" watchObservedRunningTime="2026-10-31 03:15:50.465670142 +0000 UTC m=+46.984034356" Oct 31 03:15:50.466934 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:50.466911 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:15:50.470247 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:50.470211 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954e8b0e_c426_4cee_80d6_e296726215d4.slice/crio-3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca WatchSource:0}: Error finding container 3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca: Status 404 returned error can't find the container with id 3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca Oct 31 03:15:51.257346 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:51.257309 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerStarted","Data":"3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca"} Oct 31 03:15:52.263255 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.263151 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vbv64" event={"ID":"4080580a-a99a-49e0-a255-06cb08581290","Type":"ContainerStarted","Data":"f5783a2fe01f5bb11f00cc05d5c777d1f84368ffd51c49c907f25fce17ac90a8"} Oct 31 03:15:52.265817 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.265785 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" event={"ID":"0577b36e-dd61-41bb-899d-c7af7b388696","Type":"ContainerStarted","Data":"6de3479c1b3287370232d74387361eaaf5a27cf545af408f9ffa1d73f0c8f924"} Oct 31 03:15:52.265947 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.265853 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" event={"ID":"0577b36e-dd61-41bb-899d-c7af7b388696","Type":"ContainerStarted","Data":"0baa56496b83c0a3c40759a250c74894a5ecfaacf09726d3ea2be842cd6aa11a"} Oct 31 03:15:52.265947 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.265877 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" event={"ID":"0577b36e-dd61-41bb-899d-c7af7b388696","Type":"ContainerStarted","Data":"294840d9971a38d2b712ab119e8dcbef6e02c02266c86bb90b57d4186fbcc46d"} Oct 31 03:15:52.267670 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.267643 2579 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" exitCode=0 Oct 31 03:15:52.267804 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.267726 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733"} Oct 31 03:15:52.270587 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.270564 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s8l89" event={"ID":"f09f77c2-e722-4f39-b4c9-2fcca44268a9","Type":"ContainerStarted","Data":"7a420c6fc837c085dcf87abe23f632b55bcdad641b7c53dffaef834f70ea1b88"} Oct 31 03:15:52.275196 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.275163 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" event={"ID":"75f26e8e-228a-4b68-8851-c580c827e242","Type":"ContainerStarted","Data":"45a1686766612e3079770fa9c6d7b4e01ac6088b91c1ebebee287de41e9a392e"} Oct 31 03:15:52.277153 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.277133 2579 generic.go:356] "Generic (PLEG): container finished" podID="85ec7f68-370c-4e14-a990-877b350fcdf8" containerID="08ef0113cc7f877c182ff7fbfd7cda54019e6d6bec02247fe681a95d723661a5" exitCode=0 Oct 31 03:15:52.277274 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.277237 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsvct" event={"ID":"85ec7f68-370c-4e14-a990-877b350fcdf8","Type":"ContainerDied","Data":"08ef0113cc7f877c182ff7fbfd7cda54019e6d6bec02247fe681a95d723661a5"} Oct 31 03:15:52.317513 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.317466 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vbv64" podStartSLOduration=2.027060472 podStartE2EDuration="6.317452091s" totalImagesPullingTime="4.290391619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:52.282777438 +0000 UTC m=+48.801141650" watchObservedRunningTime="2026-10-31 03:15:52.317452091 +0000 UTC m=+48.835816301" Oct 31 03:15:52.338987 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.338843 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" podStartSLOduration=2.7223941419999997 podStartE2EDuration="4.338831179s" totalImagesPullingTime="1.616437037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:52.338581501 +0000 UTC m=+48.856945712" watchObservedRunningTime="2026-10-31 03:15:52.338831179 +0000 UTC m=+48.857195387" Oct 31 03:15:52.387231 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:52.385713 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" podStartSLOduration=2.584731175 podStartE2EDuration="4.385696274s" totalImagesPullingTime="1.800965099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:52.385075248 +0000 UTC m=+48.903439544" watchObservedRunningTime="2026-10-31 03:15:52.385696274 +0000 UTC m=+48.904060487" Oct 31 03:15:53.187661 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.187594 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s8l89" podStartSLOduration=3.051895087 podStartE2EDuration="7.187574883s" totalImagesPullingTime="4.135679796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:52.411425811 +0000 UTC m=+48.929790023" watchObservedRunningTime="2026-10-31 03:15:53.187574883 +0000 UTC m=+49.705939093" Oct 31 03:15:53.187995 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.187948 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d97946f89-wfdp6"] Oct 31 03:15:53.191509 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.191474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.199793 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.199751 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 03:15:53.201022 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.200998 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 03:15:53.207296 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.207263 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d97946f89-wfdp6"] Oct 31 03:15:53.229515 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.229460 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rpnsw\"" Oct 31 03:15:53.229515 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.229460 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 03:15:53.229710 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.229554 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 03:15:53.229710 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.229636 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a61rlu4tvuhve\"" Oct 31 03:15:53.283066 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.282961 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsvct" event={"ID":"85ec7f68-370c-4e14-a990-877b350fcdf8","Type":"ContainerStarted","Data":"12cc2b426ff7850c8df70298832040d9253b43564f72e4afacd6110b07cf8c3a"} Oct 31 03:15:53.283066 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.283030 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gsvct" event={"ID":"85ec7f68-370c-4e14-a990-877b350fcdf8","Type":"ContainerStarted","Data":"e7f715f91144ad295c1a3dd7e35a45d0326f3c0a04fecc898af4b13a17dfc984"} Oct 31 03:15:53.293361 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.293337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb5cd4aa-cca0-44d0-b314-05ed0788d784-audit-log\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.293540 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.293369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.293540 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.293393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-metrics-server-audit-profiles\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.293540 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.293457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkpf\" (UniqueName: \"kubernetes.io/projected/eb5cd4aa-cca0-44d0-b314-05ed0788d784-kube-api-access-tjkpf\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.293649 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.293581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-tls\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.293649 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.293637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-client-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.293708 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.293669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-client-certs\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.305057 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.305015 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gsvct" podStartSLOduration=3.144636804 podStartE2EDuration="5.305003156s" totalImagesPullingTime="2.026252908s" totalInitContainerRuntime="134.113444ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:53.304183704 +0000 UTC m=+49.822547916" watchObservedRunningTime="2026-10-31 03:15:53.305003156 +0000 UTC m=+49.823367407" Oct 31 03:15:53.394443 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.394416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-client-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.394542 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.394504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-client-certs\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.394744 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.394725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb5cd4aa-cca0-44d0-b314-05ed0788d784-audit-log\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.394806 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.394785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.394868 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.394855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-metrics-server-audit-profiles\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.394912 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.394875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkpf\" (UniqueName: \"kubernetes.io/projected/eb5cd4aa-cca0-44d0-b314-05ed0788d784-kube-api-access-tjkpf\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.394912 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.394905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-tls\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.395924 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.395901 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb5cd4aa-cca0-44d0-b314-05ed0788d784-audit-log\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.397192 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.397162 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.397767 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.397747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-tls\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.397910 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.397887 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-metrics-server-audit-profiles\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.398663 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.398647 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-client-certs\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.399508 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.399488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-client-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.408100 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.408076 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkpf\" (UniqueName: \"kubernetes.io/projected/eb5cd4aa-cca0-44d0-b314-05ed0788d784-kube-api-access-tjkpf\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.506308 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.506266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:15:53.784576 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:53.784427 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d97946f89-wfdp6"] Oct 31 03:15:53.786416 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:53.786390 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5cd4aa_cca0_44d0_b314_05ed0788d784.slice/crio-0fb1feac279d8012749156eab7917b1d0038ccef6398e10464ee317360be60d9 WatchSource:0}: Error finding container 0fb1feac279d8012749156eab7917b1d0038ccef6398e10464ee317360be60d9: Status 404 returned error can't find the container with id 0fb1feac279d8012749156eab7917b1d0038ccef6398e10464ee317360be60d9 Oct 31 03:15:54.286942 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:54.286905 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" event={"ID":"eb5cd4aa-cca0-44d0-b314-05ed0788d784","Type":"ContainerStarted","Data":"0fb1feac279d8012749156eab7917b1d0038ccef6398e10464ee317360be60d9"} Oct 31 03:15:54.288403 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:54.288382 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc4df878-chnw2" event={"ID":"0fbe170d-bfcb-4e38-8182-59fdcb8b1628","Type":"ContainerStarted","Data":"ab86b2318db991b58936225bbb5646ea6e8deb441833d5ac1655cfc7881b6f7c"} Oct 31 03:15:54.307226 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:54.307156 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66fc4df878-chnw2" podStartSLOduration=2.348383965 podStartE2EDuration="6.30714369s" totalImagesPullingTime="3.958759725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:54.30624344 +0000 UTC m=+50.824607653" watchObservedRunningTime="2026-10-31 03:15:54.30714369 +0000 UTC m=+50.825507902" Oct 31 03:15:55.913737 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:55.913700 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f4c575df7-h42hr"] Oct 31 03:15:55.916748 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:55.916726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:55.927566 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:55.927531 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4c575df7-h42hr"] Oct 31 03:15:55.934479 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:55.934452 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 03:15:56.021840 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.021805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.022041 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.021873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.022041 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.021980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.022041 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.022020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.022183 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.022091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.022183 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.022117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.022183 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.022144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxcr\" (UniqueName: \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.022183 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.022160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123245 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123245 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123424 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123424 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123424 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123424 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxcr\" (UniqueName: \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123576 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123576 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.123994 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.123935 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.124088 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.124014 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.124088 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.124032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.124323 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.124303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.124459 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.124439 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.125849 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.125832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.125886 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.125858 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.132049 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.132028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxcr\" (UniqueName: \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.228867 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.228834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:15:56.306083 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.306053 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerStarted","Data":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} Oct 31 03:15:56.306248 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.306119 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerStarted","Data":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} Oct 31 03:15:56.306248 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.306144 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerStarted","Data":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} Oct 31 03:15:56.306248 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.306167 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerStarted","Data":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} Oct 31 03:15:56.306248 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.306188 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerStarted","Data":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} Oct 31 03:15:56.309507 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.309484 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" event={"ID":"eb5cd4aa-cca0-44d0-b314-05ed0788d784","Type":"ContainerStarted","Data":"0bade7566f441a90cdd9c43693ba6d136dd7777479dbbc7c5a0ff758485ca0b7"} Oct 31 03:15:56.329572 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.329525 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" podStartSLOduration=1.298768259 podStartE2EDuration="3.329510842s" totalImagesPullingTime="2.030742583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:56.328577942 +0000 UTC m=+52.846942157" watchObservedRunningTime="2026-10-31 03:15:56.329510842 +0000 UTC m=+52.847875114" Oct 31 03:15:56.518250 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:56.518229 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4c575df7-h42hr"] Oct 31 03:15:56.522258 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:15:56.522228 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44a09c8_5e56_478e_b27f_eca60624a37b.slice/crio-c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554 WatchSource:0}: Error finding container c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554: Status 404 returned error can't find the container with id c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554 Oct 31 03:15:57.316482 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:57.316449 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerStarted","Data":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} Oct 31 03:15:57.318006 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:57.317978 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4c575df7-h42hr" event={"ID":"a44a09c8-5e56-478e-b27f-eca60624a37b","Type":"ContainerStarted","Data":"1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4"} Oct 31 03:15:57.318112 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:57.318020 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4c575df7-h42hr" event={"ID":"a44a09c8-5e56-478e-b27f-eca60624a37b","Type":"ContainerStarted","Data":"c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554"} Oct 31 03:15:57.349931 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:57.349878 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.926502456 podStartE2EDuration="8.34986161s" totalImagesPullingTime="5.410790385s" totalInitContainerRuntime="12.568769ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:57.347548749 +0000 UTC m=+53.865912961" watchObservedRunningTime="2026-10-31 03:15:57.34986161 +0000 UTC m=+53.868225823" Oct 31 03:15:57.371872 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:57.371822 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f4c575df7-h42hr" podStartSLOduration=2.371806487 podStartE2EDuration="2.371806487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:57.370711807 +0000 UTC m=+53.889076019" watchObservedRunningTime="2026-10-31 03:15:57.371806487 +0000 UTC m=+53.890170787" Oct 31 03:15:59.127310 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:59.127277 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:59.127740 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:59.127358 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:59.133018 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:59.132994 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:15:59.189995 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:59.189969 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:15:59.329716 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:15:59.329693 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:01.280179 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:01.280151 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:06.229719 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:06.229689 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:06.230093 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:06.229749 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:06.234433 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:06.234413 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:06.352503 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:06.352478 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:06.414780 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:06.414747 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fc4df878-chnw2"] Oct 31 03:16:09.726823 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.726789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:16:09.727319 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.726842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:09.727319 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.726864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:16:09.732668 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.732651 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:16:09.733737 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.733721 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:16:09.733790 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.733739 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:16:09.740180 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.740155 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:09.740315 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.740191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:16:09.740986 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.740969 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:16:09.750454 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.750426 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:16:09.771705 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.771672 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\"" Oct 31 03:16:09.774714 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.774696 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\"" Oct 31 03:16:09.778802 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.778787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:16:09.782024 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.782005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:16:09.792683 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.792662 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\"" Oct 31 03:16:09.800281 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:09.799702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:10.056140 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:10.056116 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dlvjw"] Oct 31 03:16:10.058497 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:16:10.058474 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1ec6c2_6309_4e90_ae0b_b85aefc8c637.slice/crio-84c49a65b45392ba094933e31963b0d46ca4c4e5988749fa74e9e91a4750ae64 WatchSource:0}: Error finding container 84c49a65b45392ba094933e31963b0d46ca4c4e5988749fa74e9e91a4750ae64: Status 404 returned error can't find the container with id 84c49a65b45392ba094933e31963b0d46ca4c4e5988749fa74e9e91a4750ae64 Oct 31 03:16:10.068411 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:10.068384 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flc2b"] Oct 31 03:16:10.071602 ip-10-0-141-153 kubenswrapper[2579]: W1031 03:16:10.071572 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c43876_cf34_43bb_b641_01e1b6a4b143.slice/crio-e9340065a4ce11787596a30eea7a6ca8d13d0abafb211b971194225549c921ca WatchSource:0}: Error finding container e9340065a4ce11787596a30eea7a6ca8d13d0abafb211b971194225549c921ca: Status 404 returned error can't find the container with id e9340065a4ce11787596a30eea7a6ca8d13d0abafb211b971194225549c921ca Oct 31 03:16:10.086466 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:10.086447 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9cczw"] Oct 31 03:16:10.361730 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:10.361648 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dlvjw" event={"ID":"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637","Type":"ContainerStarted","Data":"84c49a65b45392ba094933e31963b0d46ca4c4e5988749fa74e9e91a4750ae64"} Oct 31 03:16:10.362758 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:10.362737 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9cczw" event={"ID":"73113784-937f-492c-9ac1-1a98c59d43e6","Type":"ContainerStarted","Data":"e6d68cdb634edd3bc67b85ae3cf81b484336f01ba6213d095188543c25c0c1d8"} Oct 31 03:16:10.363794 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:10.363768 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flc2b" event={"ID":"d9c43876-cf34-43bb-b641-01e1b6a4b143","Type":"ContainerStarted","Data":"e9340065a4ce11787596a30eea7a6ca8d13d0abafb211b971194225549c921ca"} Oct 31 03:16:12.375878 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:12.375842 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flc2b" event={"ID":"d9c43876-cf34-43bb-b641-01e1b6a4b143","Type":"ContainerStarted","Data":"c96cbddf650ad288aa4708d44893c9f8cea9ad99a1f479d39eaa42acd1316a09"} Oct 31 03:16:12.376342 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:12.375897 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flc2b" event={"ID":"d9c43876-cf34-43bb-b641-01e1b6a4b143","Type":"ContainerStarted","Data":"de1a4d786fe2f6a45b0cb23452c5e25ac49638a6e17e61ab533d0d57111ff9e6"} Oct 31 03:16:12.395485 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:12.395420 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-flc2b" podStartSLOduration=67.21353364 podStartE2EDuration="1m8.395403731s" totalImagesPullingTime="1.181870091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:12.395113102 +0000 UTC m=+68.913477314" watchObservedRunningTime="2026-10-31 03:16:12.395403731 +0000 UTC m=+68.913767943" Oct 31 03:16:13.506948 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:13.506923 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:13.507341 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:13.506978 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:14.388020 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:14.387978 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dlvjw" event={"ID":"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637","Type":"ContainerStarted","Data":"7f8e24ca7353c41e34d10ac31aa43a4e3a578f069e48b911cbb49b1a1464fa5c"} Oct 31 03:16:14.390081 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:14.390033 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:16:14.411340 ip-10-0-141-153 kubenswrapper[2579]: I1031 03:16:14.411244 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dlvjw" podStartSLOduration=67.015226961 podStartE2EDuration="1m10.41122607s" totalImagesPullingTime="3.395999109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:14.41049004 +0000 UTC m=+70.928854264" watchObservedRunningTime="2026-10-31 03:16:14.41122607 +0000 UTC m=+70.929590280" Oct 31 03:16:15.736425 ip-10-0-141-153 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 03:16:15.757774 ip-10-0-141-153 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 03:16:15.758042 ip-10-0-141-153 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 03:16:15.758330 ip-10-0-141-153 systemd[1]: kubelet.service: Consumed 4.195s CPU time. Oct 31 03:16:15.772975 ip-10-0-141-153 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:16:15.833746 ip-10-0-141-153 kubenswrapper[5520]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:16:15.833746 ip-10-0-141-153 kubenswrapper[5520]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:16:15.833746 ip-10-0-141-153 kubenswrapper[5520]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:16:15.833746 ip-10-0-141-153 kubenswrapper[5520]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:16:15.835779 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835698 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.835779 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835780 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.835779 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835783 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835786 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835789 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835792 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835795 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835797 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835800 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835802 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835805 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835807 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835810 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835813 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835816 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835818 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835821 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835823 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835826 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835828 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835831 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.835880 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835834 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835836 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835839 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835842 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835844 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835847 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835849 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835857 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835860 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835862 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835865 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835868 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835870 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835873 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835875 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835878 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835881 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835884 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835886 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835889 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.836440 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835892 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835895 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835897 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835900 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835902 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835905 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835908 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835910 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835913 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835915 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835918 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835923 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835925 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835928 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835930 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835933 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835936 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835938 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835941 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835944 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835947 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.837284 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835949 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835952 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835954 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835957 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835959 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835962 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835965 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835968 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835970 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835973 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835977 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835980 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835982 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835984 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835987 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835989 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835992 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835994 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835997 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.835999 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.837971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836002 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836004 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836008 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836011 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836013 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836016 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836018 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836021 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836023 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836026 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836028 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836031 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836035 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836039 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836042 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836047 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836052 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836055 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.838507 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836058 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836060 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836063 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836065 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836068 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.836070 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838091 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838103 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838106 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838109 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838113 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838116 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838118 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838121 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838124 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838126 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838129 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838133 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838139 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.838970 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838143 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838145 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838148 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838151 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838153 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838156 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838159 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838161 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838164 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838166 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838169 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838172 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838174 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838177 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838180 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838183 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838185 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838188 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838191 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838194 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.839480 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838215 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838220 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838224 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838227 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838230 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838232 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838235 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838238 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838241 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838244 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838246 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838249 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838252 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838254 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838257 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838260 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838262 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838269 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838272 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838275 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.839984 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838277 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838280 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838284 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838286 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838288 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838292 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838294 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838297 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838299 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838302 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838304 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838308 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838310 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838313 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838315 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838318 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838320 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838323 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838325 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.840513 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838328 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838330 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838334 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838339 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838341 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838344 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838347 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838349 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838351 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838354 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838357 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838360 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838363 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838365 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838368 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838371 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838373 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838376 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838379 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.841018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838381 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838384 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838386 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838389 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838391 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838394 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838397 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838400 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838404 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838406 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838408 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838411 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838413 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838416 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838418 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838912 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838917 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838919 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838922 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.841493 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838925 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838927 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838929 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838932 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838935 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838938 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838941 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838945 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838948 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838952 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838954 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838957 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838959 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838962 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838965 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838967 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838970 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838973 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838975 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838978 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.842050 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838980 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838983 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838986 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838988 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838991 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838993 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838995 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.838998 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839001 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839003 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839006 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839008 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839011 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839013 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839015 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839018 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839021 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839032 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839036 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839039 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.842582 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839041 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839044 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839047 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839049 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839052 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839055 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839058 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839061 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839064 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839066 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839069 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839071 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839074 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839076 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839079 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839082 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839084 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839087 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839089 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.843084 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839092 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839095 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839097 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839101 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839104 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839107 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839110 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839112 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839115 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839117 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839120 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839123 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839127 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839130 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839132 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839135 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839138 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839140 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839143 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.843611 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839146 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839148 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839151 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839153 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839155 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839158 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839161 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839163 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839166 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839169 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839171 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839174 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839176 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839179 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839181 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839184 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839187 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839189 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839192 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.844080 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839194 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839221 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839225 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839227 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839229 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839323 5520 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839334 5520 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839341 5520 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839346 5520 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839350 5520 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839354 5520 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839358 5520 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839361 5520 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839365 5520 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839368 5520 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839371 5520 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839374 5520 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839377 5520 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839380 5520 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839383 5520 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839388 5520 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839390 5520 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839394 5520 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:16:15.844575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839398 5520 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839401 5520 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839404 5520 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839407 5520 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839410 5520 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839414 5520 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839417 5520 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839420 5520 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839423 5520 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839425 5520 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839428 5520 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839431 5520 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839434 5520 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839437 5520 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839441 5520 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839444 5520 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839447 5520 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839450 5520 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839454 5520 flags.go:64] FLAG: --enable-server="true" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839456 5520 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839460 5520 flags.go:64] FLAG: --event-burst="100" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839463 5520 flags.go:64] FLAG: --event-qps="50" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839466 5520 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839469 5520 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:16:15.845154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839472 5520 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839476 5520 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839479 5520 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839482 5520 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839485 5520 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839492 5520 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839495 5520 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839498 5520 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839501 5520 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839503 5520 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839506 5520 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839510 5520 flags.go:64] FLAG: --feature-gates="" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839514 5520 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839517 5520 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839520 5520 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839523 5520 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839526 5520 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839529 5520 flags.go:64] FLAG: --help="false" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839531 5520 flags.go:64] FLAG: --hostname-override="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839534 5520 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839537 5520 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839540 5520 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839543 5520 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:16:15.845760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839546 5520 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839549 5520 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839552 5520 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839556 5520 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839559 5520 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839562 5520 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839564 5520 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839567 5520 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839570 5520 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839572 5520 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839576 5520 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839578 5520 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839581 5520 flags.go:64] FLAG: --lock-file="" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839584 5520 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839588 5520 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839591 5520 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839600 5520 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839603 5520 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839606 5520 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839609 5520 flags.go:64] FLAG: --logging-format="text" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839612 5520 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839615 5520 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839618 5520 flags.go:64] FLAG: --manifest-url="" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839620 5520 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839625 5520 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:16:15.846336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839628 5520 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839631 5520 flags.go:64] FLAG: --max-pods="110" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839634 5520 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839637 5520 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839639 5520 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839642 5520 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839645 5520 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839648 5520 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839651 5520 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839659 5520 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839662 5520 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839666 5520 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839670 5520 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839672 5520 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839675 5520 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839678 5520 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839681 5520 flags.go:64] FLAG: --port="10250" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839684 5520 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839686 5520 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0047b1328d2b4b254" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839689 5520 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839692 5520 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839696 5520 flags.go:64] FLAG: --register-node="true" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839699 5520 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839703 5520 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839706 5520 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839708 5520 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:16:15.846941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839711 5520 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839715 5520 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839718 5520 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839721 5520 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839723 5520 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839726 5520 flags.go:64] FLAG: --runonce="false" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839729 5520 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839732 5520 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839735 5520 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839738 5520 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839741 5520 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839744 5520 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839747 5520 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839750 5520 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839752 5520 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839755 5520 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839758 5520 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839761 5520 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839764 5520 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839767 5520 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839770 5520 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839775 5520 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839777 5520 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839780 5520 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839785 5520 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:16:15.847688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839787 5520 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839790 5520 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839794 5520 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839797 5520 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839800 5520 flags.go:64] FLAG: --v="2" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839804 5520 flags.go:64] FLAG: --version="false" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839807 5520 flags.go:64] FLAG: --vmodule="" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839811 5520 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.839815 5520 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839959 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839963 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839967 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839970 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839973 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839976 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839978 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839981 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839984 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839987 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839989 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839992 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839994 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.848302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.839998 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840000 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840003 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840006 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840010 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840013 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840016 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840019 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840021 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840024 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840026 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840029 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840032 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840035 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840037 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840040 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840042 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840045 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840047 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.848888 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840050 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840052 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840055 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840057 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840060 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840062 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840065 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840067 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840070 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840072 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840075 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840077 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840081 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840084 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840086 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840088 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840091 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840093 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840096 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840099 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.849400 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840101 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840104 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840106 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840109 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840111 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840113 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840116 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840120 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840124 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840127 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840129 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840132 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840134 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840136 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840139 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840142 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840144 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840147 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840149 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.850089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840152 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840154 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840156 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840159 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840161 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840165 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840168 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840170 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840173 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840175 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840177 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840180 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840183 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840186 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840189 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840191 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840193 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840196 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840242 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840245 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.850689 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840248 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840250 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840253 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840255 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840258 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840260 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840263 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840265 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840268 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840271 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840273 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840276 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840278 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840281 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840283 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.840309 5520 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:16:15.851186 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840357 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840363 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840367 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840370 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840373 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840375 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840378 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840381 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840383 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840386 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840388 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840391 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840393 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840396 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840398 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840402 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840404 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840407 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840410 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.851659 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840412 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840415 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840417 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840420 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840422 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840425 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840427 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840430 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840432 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840435 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840437 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840440 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840442 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840445 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840450 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840453 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840455 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840458 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840460 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840463 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.852246 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840465 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840468 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840471 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840476 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840480 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840483 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840486 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840489 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840492 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840495 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840498 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840501 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840503 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840506 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840508 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840510 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840513 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840515 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840518 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.852822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840521 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840524 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840527 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840530 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840532 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840535 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840537 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840542 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840544 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840547 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840549 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840552 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840554 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840557 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840559 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840562 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840565 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840568 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840571 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840573 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.853576 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840575 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840579 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840582 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840584 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840587 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840589 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840594 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840598 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840605 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840609 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840611 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840614 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840616 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840619 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840621 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840624 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840626 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840629 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840631 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.854383 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840635 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840637 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840640 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840642 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840645 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840647 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840650 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840652 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.840655 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.844507 5520 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.844521 5520 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.844539 5520 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.844543 5520 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844601 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844606 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844609 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844613 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844616 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844618 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.854910 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844621 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844624 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844627 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844631 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844635 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844638 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844640 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844643 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844646 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844648 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844651 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844653 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844656 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844658 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844661 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844663 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844666 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844668 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844671 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844673 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.855406 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844676 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844678 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844681 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844684 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844688 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844691 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844693 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844696 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844698 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844701 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844703 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844706 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844709 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844711 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844714 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844716 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844718 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844721 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844723 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844726 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.855925 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844728 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844731 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844733 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844736 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844738 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844741 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844744 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844746 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844749 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844751 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844754 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844756 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844759 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844761 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844764 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844767 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844769 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844772 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844775 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.856701 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844777 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844780 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844782 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844785 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844787 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844790 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844792 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844795 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844798 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844800 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844803 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844805 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844808 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844812 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844815 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844818 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844820 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844823 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844825 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.857371 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844827 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844830 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844832 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844835 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844837 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844840 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844842 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844845 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844847 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844850 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844852 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844855 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844858 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844860 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844863 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844866 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844868 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844871 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844873 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.858034 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844876 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844878 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.844881 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.844907 5520 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845044 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845048 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845051 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845054 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845058 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845062 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845064 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845068 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845071 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845073 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845076 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845078 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.858542 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845081 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845083 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845086 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845088 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845091 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845093 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845096 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845098 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845101 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845103 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845106 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845109 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845111 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845116 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845118 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845121 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845124 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845126 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845129 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845131 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.858948 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845134 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845136 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845139 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845142 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845144 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845146 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845149 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845151 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845154 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845156 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845159 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845162 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845165 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845168 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845170 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845173 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845175 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845178 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845181 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.859466 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845183 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845185 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845188 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845190 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845193 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845196 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845217 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845221 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845223 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845226 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845229 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845231 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845234 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845237 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845240 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845243 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845245 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845248 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845250 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845253 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.859932 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845255 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845258 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845260 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845263 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845266 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845269 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845271 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845274 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845276 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845279 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845281 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845284 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845286 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845288 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845291 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845294 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845296 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845299 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845304 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.860459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845306 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845309 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845311 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845314 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845317 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845319 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845322 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845324 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845327 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845330 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845332 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845336 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845340 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845343 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845346 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.860913 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845348 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.845372 5520 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845406 5520 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845410 5520 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845413 5520 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845416 5520 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845419 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845422 5520 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845425 5520 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845427 5520 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845430 5520 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845432 5520 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845435 5520 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845437 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845440 5520 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845443 5520 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:15.861300 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845447 5520 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845453 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845456 5520 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845459 5520 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845462 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845464 5520 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845467 5520 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845469 5520 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845472 5520 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845476 5520 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845480 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845482 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845485 5520 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845487 5520 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845490 5520 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845492 5520 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845495 5520 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845497 5520 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845500 5520 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:15.861699 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845502 5520 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845505 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845507 5520 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845510 5520 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845512 5520 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845515 5520 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845517 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845519 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845522 5520 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845524 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845527 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845529 5520 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845532 5520 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845534 5520 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845539 5520 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845542 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845545 5520 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845547 5520 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845550 5520 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845552 5520 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:15.862163 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845555 5520 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845557 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845560 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845562 5520 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845565 5520 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845567 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845570 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845572 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845575 5520 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845577 5520 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845580 5520 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845582 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845585 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845587 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845591 5520 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845594 5520 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845596 5520 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845599 5520 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845601 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:15.862666 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845604 5520 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845606 5520 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845608 5520 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845611 5520 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845613 5520 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845616 5520 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845618 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845622 5520 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845625 5520 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845628 5520 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845631 5520 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845633 5520 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845636 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845638 5520 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845640 5520 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845643 5520 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845645 5520 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845648 5520 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845650 5520 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845652 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:15.863234 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845655 5520 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845658 5520 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845661 5520 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845663 5520 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845666 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845668 5520 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845671 5520 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845674 5520 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845677 5520 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845679 5520 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845682 5520 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845684 5520 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845686 5520 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:15.845689 5520 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.845737 5520 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.846707 5520 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.846760 5520 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.846934 5520 server.go:1014] "Starting client certificate rotation" Oct 31 03:16:15.863734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.847075 5520 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.847121 5520 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:10:04 +0000 UTC" deadline="2028-07-31 06:10:42.293743966 +0000 UTC" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.847156 5520 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15338h54m26.446590046s" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.847405 5520 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.848419 5520 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.849688 5520 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.851538 5520 log.go:25] "Validated CRI v1 image API" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.852249 5520 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:16:15.864175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.855070 5520 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bce90099-8273-4ef9-8653-de7b6659b793:/dev/nvme0n1p3 e5ebff43-e782-4b35-890a-9b8ced84f7ef:/dev/nvme0n1p4] Oct 31 03:16:15.864580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.855086 5520 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00b703925370846face39bcf6f0b1aa8b06d4c20c9cad2abdc1b7c66d8264798/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00b703925370846face39bcf6f0b1aa8b06d4c20c9cad2abdc1b7c66d8264798/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/029aaed000d33004247b040142c85c5550d1e3235981d0cc3b3a16ab9d938c40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/029aaed000d33004247b040142c85c5550d1e3235981d0cc3b3a16ab9d938c40/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08a8657a575ac593b1c963f6d728cec6c1f5337de88a5de0e0190df490623da7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08a8657a575ac593b1c963f6d728cec6c1f5337de88a5de0e0190df490623da7/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ab644eac17d3db659a617f20aeb7561c7269c3951589fcaa8b62c0872ba9b23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ab644eac17d3db659a617f20aeb7561c7269c3951589fcaa8b62c0872ba9b23/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fb1feac279d8012749156eab7917b1d0038ccef6398e10464ee317360be60d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fb1feac279d8012749156eab7917b1d0038ccef6398e10464ee317360be60d9/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15ca714cb6a6596bc767cc6f24cd305afc64c1626a52c5fc9843c5894fb72073/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15ca714cb6a6596bc767cc6f24cd305afc64c1626a52c5fc9843c5894fb72073/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31520d8049803b599e914cd7b0a04ac635442220a46039e330942f90b50efcaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31520d8049803b599e914cd7b0a04ac635442220a46039e330942f90b50efcaa/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31ff3f5f369ad136854c162dcb692d3d97059b0c2109755a452b33e64cec2b9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31ff3f5f369ad136854c162dcb692d3d97059b0c2109755a452b33e64cec2b9c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a2cbbba7fc88a2fa1c4789653fdaa6da169cfe897e952b3fb77bd103cf08455/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a2cbbba7fc88a2fa1c4789653fdaa6da169cfe897e952b3fb77bd103cf08455/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fbf68753c4f03805823cd86fc77d22f11652fafe58de6caebe1d3bb2723dcd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fbf68753c4f03805823cd86fc77d22f11652fafe58de6caebe1d3bb2723dcd0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70df0484b897b8dd05644219f0263c649060426627184180e4e2ee0dfb9d2e2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70df0484b897b8dd05644219f0263c649060426627184180e4e2ee0dfb9d2e2a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7238234ccd5cb864ec2568d80406c485b9203a4c483e6a3af0dcf0265dcb8337/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7238234ccd5cb864ec2568d80406c485b9203a4c483e6a3af0dcf0265dcb8337/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84c49a65b45392ba094933e31963b0d46ca4c4e5988749fa74e9e91a4750ae64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84c49a65b45392ba094933e31963b0d46ca4c4e5988749fa74e9e91a4750ae64/userdata/shm major:0 minor:401 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/912adab67e1d947caa2409ae9a5582fe858c2c79853d1304ed6a0b1d234532c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/912adab67e1d947caa2409ae9a5582fe858c2c79853d1304ed6a0b1d234532c9/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ba6f3486d62ff9e0862d8561b9a8aeb84fde0814a6ca9b5c07d4c8b84b954e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ba6f3486d62ff9e0862d8561b9a8aeb84fde0814a6ca9b5c07d4c8b84b954e3/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554/userdata/shm major:0 minor:383 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6d68cdb634edd3bc67b85ae3cf81b484336f01ba6213d095188543c25c0c1d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6d68cdb634edd3bc67b85ae3cf81b484336f01ba6213d095188543c25c0c1d8/userdata/shm major:0 minor:405 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9340065a4ce11787596a30eea7a6ca8d13d0abafb211b971194225549c921ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9340065a4ce11787596a30eea7a6ca8d13d0abafb211b971194225549c921ca/userdata/shm major:0 minor:402 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eebb12123ada46fdc68dc01673e2280e7e521dfe1686a72d03e382c9f231d67a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eebb12123ada46fdc68dc01673e2280e7e521dfe1686a72d03e382c9f231d67a/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~projected/kube-api-access-ngg2k:{mountpoint:/var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~projected/kube-api-access-ngg2k major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~projected/kube-api-access-j9vnm:{mountpoint:/var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~projected/kube-api-access-j9vnm major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/196bf414-2c29-40a0-a957-f94cadaa7ff2/volumes/kubernetes.io~projected/kube-api-access-vsljf:{mountpoint:/var/lib/kubelet/pods/196bf414-2c29-40a0-a957-f94cadaa7ff2/volumes/kubernetes.io~projected/kube-api-access-vsljf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3387800d-492d-4f65-b3d8-4bbccd906d03/volumes/kubernetes.io~projected/kube-api-access-pnjxn:{mountpoint:/var/lib/kubelet/pods/3387800d-492d-4f65-b3d8-4bbccd906d03/volumes/kubernetes.io~projected/kube-api-access-pnjxn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4080580a-a99a-49e0-a255-06cb08581290/volumes/kubernetes.io~projected/kube-api-access-jzrs6:{mountpoint:/var/lib/kubelet/pods/4080580a-a99a-49e0-a255-06cb08581290/volumes/kubernetes.io~projected/kube-api-access-jzrs6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4080580a-a99a-49e0-a255-06cb08581290/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4080580a-a99a-49e0-a255-06cb08581290/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453eddca-0701-4325-b5e2-12b4e3e78565/volumes/kubernetes.io~projected/kube-api-access-k4p9n:{mountpoint:/var/lib/kubelet/pods/453eddca-0701-4325-b5e2-12b4e3e78565/volumes/kubernetes.io~projected/kube-api-access-k4p9n major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volumes/kubernetes.io~projected/kube-api-access-72bmk:{mountpoint:/var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volumes/kubernetes.io~projected/kube-api-access-72bmk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637/volumes/kubernetes.io~projected/kube-api-access-kfm56:{mountpoint:/var/lib/kubelet/pods/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637/volumes/kubernetes.io~projected/kube-api-access-kfm56 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73113784-937f-492c-9ac1-1a98c59d43e6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/73113784-937f-492c-9ac1-1a98c59d43e6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~projected/kube-api-access-srtz9:{mountpoint:/var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~projected/kube-api-access-srtz9 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~projected/kube-api-access-ffd55:{mountpoint:/var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~projected/kube-api-access-ffd55 major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6a1947-49a2-44d8-9db3-5004938650e6/volumes/kubernetes.io~projected/kube-api-access-sk9db:{mountpoint:/var/lib/kubelet/pods/8e6a1947-49a2-44d8-9db3-5004938650e6/volumes/kubernetes.io~projected/kube-api-access-sk9db major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/alertmanager/9 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/config-reloader/6 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~empty-dir/config-out major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~projected/kube-api-access-hdkfz:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~projected/kube-api-access-hdkfz major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~projected/tls-assets major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/config-volume major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/web-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~projected/kube-api-access-5pxcr:{mountpoint:/var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~projected/kube-api-access-5pxcr major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c43876-cf34-43bb-b641-01e1b6a4b143/volumes/kubernetes.io~projected/kube-api-access-xvzf2:{mountpoint:/var/lib/kubelet/pods/d9c43876-cf34-43bb-b641-01e1b6a4b143/volumes/kubernetes.io~projected/kube-api-access-xvzf2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9c43876-cf34-43bb-b641-01e1b6a4b143/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d9c43876-cf34-43bb-b641-01e1b6a4b143/volumes/kubernetes.io~secret/metrics-certs major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df52d3e9-f738-4aaf-b8e4-de32943b6b03/volumes/kubernetes.io~projected/kube-api-access-dwvlv:{mountpoint:/var/lib/kubelet/pods/df52d3e9-f738-4aaf-b8e4-de32943b6b03/volumes/kubernetes.io~projected/kube-api-access-dwvlv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5deb6a1-16c9-4491-8e07-d8deafd8607f/volumes/kubernetes.io~projected/kube-api-access-hbgtd:{mountpoint:/var/lib/kubelet/pods/e5deb6a1-16c9-4491-8e07-d8deafd8607f/volumes/kubernetes.io~projected/kube-api-access-hbgtd major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5deb6a1-16c9-4491-8e07-d8deafd8607f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e5deb6a1-16c9-4491-8e07-d8deafd8607f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~projected/kube-api-access-tjkpf:{mountpoint:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~projected/kube-api-access-tjkpf major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~projected/kube-api-access-8ftdc:{mountpoint:/var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~projected/kube-api-access-8ftdc major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09f77c2-e722-4f39-b4c9-2fcca44268a9/volumes/kubernetes.io~projected/kube-api-access-kc8z5:{mountpoint:/var/lib/kubelet/pods/f09f77c2-e722-4f39-b4c9-2fcca44268a9/volumes/kubernetes.io~projected/kube-api-access-kc8z5 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f09f77c2-e722-4f39-b4c9-2fcca44268a9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f09f77c2-e722-4f39-b4c9-2fcca44268a9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f454b995-5cf2-461d-b6f1-932bd4a45110/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f454b995-5cf2-461d-b6f1-932bd4a45110/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f771de0f-1ffa-4de2-823d-40d0a95eed5c/volumes/kubernetes.io~projected/kube-api-access-8t2mm:{mountpoint:/var/lib/kubelet/pods/f771de0f-1ffa-4de2-823d-40d0a95eed5c/volumes/kubernetes.io~projected/kube-api-access-8t2mm major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5fa6409f961a743fd00ba486cd1d6ebf8977dda836a2df162a9861b377a77acb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5f6c2a9a8da36deaee83e8824e83ff9ff894b74a9bfba3711b8d78acd32a4de6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5a3fd578465986d18016d1e2856cf32711bcce691a5a971c560432d699a859f6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/32e9eba3d25d63d7efae57af4128d4c7ed856d69173be339091113c9abd57650/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9deebf99dd4a54eaac0614d5ba7685d1c48dee71489ec6858413e75c470797fb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cf331645bf93ed54f862c6e9b9999b598d7a85bda8d24049eb29110bf49dc1b5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/09b86d957252511b0f700262202d4a3a8c64266b2a92f81dc97236ad1697ae89/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ed0e9cc0044d91c85db34c73b7c6ce9cd56cec2790e6b5971f7e8acb6ac5b603/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6088ab05a70fe3c3954bb8b8da4f819f66746b5788284086d99e41478235e83d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/300bf4f85d079fd8c4a154b4aab4952bbf809cb657c4bae1c89b3c7b470b4a2e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/bfe044511b7ecffdda090dc544a5b79c9f121663c2cb80694c1f52187d05c847/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/402a5e8f22f6f39b1ae9475e218b2e5bb6e11155c99cf657a70f472494d87ccb/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d75c73637e645156d4dade5809ea0ed066c322370ed6c8076b3c4569d6969ce7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4b5b79e1ff6209ab6fcb34408ce71e79dce47e6db5f945017f16602eb883aa8b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a24f533e5e91af2d41dac1a91ea1dbd3de61a3db36efe3f7b17f4b2235e5db43/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/eb4d26abb6eace79a96d69cbd1a1151f31c053cabf652cda73c276f81bc6d78e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/99f5e51423a5a41dbfc217cbc992b356a5d7426dc4162de90fb176152f7bd833/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a18b8688717d24acf49eedfd0c04f54630821e4c6ef94365d5f200fbb66e7cc2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2d11511595f58ce448f688d48a6549cb15489dcff682528dc56e1030ac8349c3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/224845a4127818aac768fcbe81ec8601413ad61133caefb638073370d0360a7f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a807b47fedade3aeed062daff6ff6f9f9478b4ea4a4debceb73899ab3450b1b5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8a33cffb3853435e9b000a123b0f824d7d4fe785ea5e2a15720e2f096637207e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f324f71cf485b34d11d81ede404376c6d0b5c6c71f7f438591f77564c073f7b5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/bab97d88e3092284a92bc150a3019421f5c3cf213205ba66ce1a902885cdf373/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/8ea1d326fc05f17d3aed5fa24ccdf87ad1e36c11f8d67b74c7d14be3458bd551/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/d0465a2847c49efc2945b29e76d029c9274ab9bfdcc48706b5cd523214b18215/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/682e84374a7185cbc41e0f1b9db97abdaac45970c0142d1ee3951e3bf8d2647c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f7c1183f307550f3b1d2eb9a0ee1214a76f71ac30132961d0e9cabbc2f82239b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/7a1db203497b0560f8af4d7d97fe2a9ec02e39f4dd07e291cdda70949af1cb13/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/719e0c66506685516c8ceefeed361f0e3b40a33a58906941570af3728535aba4/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/e4bf1a89cce55c2d0ae6711ffc4a299fa52d49a83a746b770630b306edb981d7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/e3f5f3ecffe2ac582401bba91cefe9fb1a15d1f92f41ab73f424deb3684265aa/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/46d64005b5ff1a7d8e243b661a53a30ab68fd1f24df7ff13f3d1ec023af7b5c2/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/0e335443dd52cb1fc29bf7c232e0be28759226d83885f0783ccc6506de63980d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/c943f68ec55a31d9f9c788bd2dd53ac38c2530e3d3724a4909fb1294312007c6/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/e5b23d031a4b30a97047e898560008e034dc43d92fa32392f5697d9e522d3a9b/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/adb532d2efcade6c10b6455deee4244f1b911ac8b7fdc0ec4c3ba93116c4911f/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/9e8c75a4a6aa43a5470216bb2611fef64c613c1c1ab78384385e52491a8a83f7/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/0acbbf91435c6b91cf5bd108c8d53cd35bbbc030924fae72d9c36acd1a12bf5b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/50c0fd2655881a39a6d13721b02f9194abf74b1472d7c2f37834dca98eaffaaf/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/1fd9256939858f8a80dacd8d83c54594040a2cc73995135c28b13cace4579c81/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a5f1b904e0998ca0cebf47e3438ebc80c95bdfcaa104174c163eec527054c214/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/7877053f1a9fa43fc33c81b00d8d5e06fe96f6f0acc50c927cf85e28e3b463b6/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/bc9e32bb678b8bb208bc08dcbb57ace5a0b7a5823bf79d3705baab906f0d4e55/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/ff639c4673f125bf525d8f2636938a673fbf8ea3e87ca5b50c94e5f2dfabbb96/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/6ad368ebdd2e6532665b788c52d7d0e4258736a79ea5fa9732b5707b3c905228/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/7dd768cbed5fd61b48c09503c9c64de3eda31bd082a09e75dd68daa30f03c8da/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/6a7978b643b8afc205d135d3a8152b9aff49292c8f4680d944043011d876231b/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/c625ba7c8a04b19455186375394a491c80a1c70b4f0d9a55e2a9b81a96a6323e/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/9902693e3e14836bb3d20a103eba62f02ecdee4bee41620ec2a84ea21f74acc1/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/74908fb4f76634b150f845def28854947ed622200c1116de95c532f3f6d6da50/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/a539ae13ad487a7ccb3e3857f9c93af422fe6df6428ce931e67fed105856817b/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/3a3e5ed18f31fa8f0b105f165ec3b142e1a1b892c1d821c3ac1d362dfe2912a4/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/42496d9733b6185371810e2cf8d674cca71d998b2fa76a9cc5d2b859da9f5de1/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/6b467f53006a0527fbec6bac7048e8afa3673f885b039c825f00b2e909caff33/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/6e41569c9417bd41709dd745202391d7e3f9766bb5da6ed6e9a6884ebb0e305c/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/9820a612f588a6122f630bbcb2b03ce067d0f18120df6c41fd05b0265c2eb41c/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/d33a95c93333908b067b04192c9e678de1e9e1ea1af91e41605144ef2c48c1d4/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/4790624a40deeb34108a4472946e9a18d4973cf2bf2d203fc00338e2e6587420/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/34f45cae40b4ffc0de69e6338aa20bf250c4d3e6050b4ae887b50b9654ccdb37/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/35ce142be4a818f4d5c5bb1096d8e8efa64d5a815ac51c75c9d093b7cab2b2bc/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/cec7ebcbd05fa36519353b82aab198e89bdab93d2f951461b2b154b941fcb3bf/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6d04f0e3f1677def9062950be9b332244537bda3b706696b58d06fe45cf6f687/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/289c50741283a86d59061b0a798013491abb140e66ac4043f4c5dfe000b798b7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/caf2df6f3f70e6aa92a8fdc1e8ce05cc705b5ab1db9d78fa0232242558f51b13/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1a0869cea9a45815847b4c80207bbb18221549d164685fcdeb796735cc9a536e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/852db7d6eb0063dd645742cbb8e279d149c16c8ff9d62bcedf78de68e85a08c7/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/260bd7d0c40a326aa0c67c428ce1267442963b634469c8ea4460e2d4f755e79f/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/aeea6ea9b8698cac32143a14190a33467f6f29057318bcd06e9580291e9ce3f8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ce302297b24a24812f8f98df33f54cddef79250b71d0df60967849a4854c4e50/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/90d0bf64f9836551f8f2c098c7421a913de24bb7ea51420a545861842be41969/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.862926 5520 manager.go:217] Machine: {Timestamp:2026-10-31 03:16:15.862471824 +0000 UTC m=+0.056994936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102143 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d870904cd2693df0b75f66f996a52 SystemUUID:ec2d8709-04cd-2693-df0b-75f66f996a52 BootID:80ce1c91-54b2-459c-9258-02af9145d80e Filesystems:[{Device:/run/containers/storage/overlay-containers/0ab644eac17d3db659a617f20aeb7561c7269c3951589fcaa8b62c0872ba9b23/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08a8657a575ac593b1c963f6d728cec6c1f5337de88a5de0e0190df490623da7/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~projected/kube-api-access-ngg2k DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4080580a-a99a-49e0-a255-06cb08581290/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eebb12123ada46fdc68dc01673e2280e7e521dfe1686a72d03e382c9f231d67a/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ba6f3486d62ff9e0862d8561b9a8aeb84fde0814a6ca9b5c07d4c8b84b954e3/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5deb6a1-16c9-4491-8e07-d8deafd8607f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:283 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:337 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4080580a-a99a-49e0-a255-06cb08581290/volumes/kubernetes.io~projected/kube-api-access-jzrs6 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c43876-cf34-43bb-b641-01e1b6a4b143/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:399 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9340065a4ce11787596a30eea7a6ca8d13d0abafb211b971194225549c921ca/userdata/shm DeviceMajor:0 DeviceMinor:402 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:234 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:336 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:284 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:338 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84c49a65b45392ba094933e31963b0d46ca4c4e5988749fa74e9e91a4750ae64/userdata/shm DeviceMajor:0 DeviceMinor:401 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554/userdata/shm DeviceMajor:0 DeviceMinor:383 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/912adab67e1d947caa2409ae9a5582fe858c2c79853d1304ed6a0b1d234532c9/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fbf68753c4f03805823cd86fc77d22f11652fafe58de6caebe1d3bb2723dcd0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73113784-937f-492c-9ac1-1a98c59d43e6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:395 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31ff3f5f369ad136854c162dcb692d3d97059b0c2109755a452b33e64cec2b9c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70df0484b897b8dd05644219f0263c649060426627184180e4e2ee0dfb9d2e2a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f09f77c2-e722-4f39-b4c9-2fcca44268a9/volumes/kubernetes.io~projected/kube-api-access-kc8z5 DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/196bf414-2c29-40a0-a957-f94cadaa7ff2/volumes/kubernetes.io~projected/kube-api-access-vsljf DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6a1947-49a2-44d8-9db3-5004938650e6/volumes/kubernetes.io~projected/kube-api-access-sk9db DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/453eddca-0701-4325-b5e2-12b4e3e78565/volumes/kubernetes.io~projected/kube-api-access-k4p9n DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:376 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:381 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~projected/kube-api-access-srtz9 DeviceMajor:0 DeviceMinor:230 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:281 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f454b995-5cf2-461d-b6f1-932bd4a45110/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volumes/kubernetes.io~projected/kube-api-access-72bmk DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb5cd4aa-cca0-44d0-b314-05ed0788d784/volumes/kubernetes.io~projected/kube-api-access-tjkpf DeviceMajor:0 DeviceMinor:339 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7238234ccd5cb864ec2568d80406c485b9203a4c483e6a3af0dcf0265dcb8337/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:280 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~projected/kube-api-access-hdkfz DeviceMajor:0 DeviceMinor:285 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f771de0f-1ffa-4de2-823d-40d0a95eed5c/volumes/kubernetes.io~projected/kube-api-access-8t2mm DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f09f77c2-e722-4f39-b4c9-2fcca44268a9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5deb6a1-16c9-4491-8e07-d8deafd8607f/volumes/kubernetes.io~projected/kube-api-access-hbgtd DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes/kubernetes.io~projected/kube-api-access-5pxcr DeviceMajor:0 DeviceMinor:382 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/df52d3e9-f738-4aaf-b8e4-de32943b6b03/volumes/kubernetes.io~projected/kube-api-access-dwvlv DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/029aaed000d33004247b040142c85c5550d1e3235981d0cc3b3a16ab9d938c40/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15ca714cb6a6596bc767cc6f24cd305afc64c1626a52c5fc9843c5894fb72073/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f07e2583-7e4e-49ef-8980-e76733289e46/volumes/kubernetes.io~projected/kube-api-access-8ftdc DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f26e8e-228a-4b68-8851-c580c827e242/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:275 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6d68cdb634edd3bc67b85ae3cf81b484336f01ba6213d095188543c25c0c1d8/userdata/shm DeviceMajor:0 DeviceMinor:405 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0577b36e-dd61-41bb-899d-c7af7b388696/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31520d8049803b599e914cd7b0a04ac635442220a46039e330942f90b50efcaa/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:279 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fb1feac279d8012749156eab7917b1d0038ccef6398e10464ee317360be60d9/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a2cbbba7fc88a2fa1c4789653fdaa6da169cfe897e952b3fb77bd103cf08455/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/5ab0c10c-8503-43e7-8892-894c3c0650d4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~projected/kube-api-access-ffd55 DeviceMajor:0 DeviceMinor:243 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:284 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:231 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:233 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:282 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3387800d-492d-4f65-b3d8-4bbccd906d03/volumes/kubernetes.io~projected/kube-api-access-pnjxn DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00b703925370846face39bcf6f0b1aa8b06d4c20c9cad2abdc1b7c66d8264798/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9c43876-cf34-43bb-b641-01e1b6a4b143/volumes/kubernetes.io~projected/kube-api-access-xvzf2 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes/kubernetes.io~projected/kube-api-access-j9vnm DeviceMajor:0 DeviceMinor:235 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/85ec7f68-370c-4e14-a990-877b350fcdf8/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:241 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637/volumes/kubernetes.io~projected/kube-api-access-kfm56 DeviceMajor:0 DeviceMinor:400 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:029aaed000d3300 MacAddress:8e:ef:35:80:f7:f5 Speed:10000 Mtu:8901} {Name:0fb1feac279d801 MacAddress:ea:b1:51:bd:2c:cf Speed:10000 Mtu:8901} {Name:31520d8049803b5 MacAddress:ae:d6:d9:03:fd:ca Speed:10000 Mtu:8901} {Name:3da2874dc6071ab MacAddress:d6:ea:68:97:e6:97 Speed:10000 Mtu:8901} {Name:4a2cbbba7fc88a2 MacAddress:b2:43:b3:d8:50:e5 Speed:10000 Mtu:8901} {Name:7238234ccd5cb86 MacAddress:3a:35:b8:9e:75:41 Speed:10000 Mtu:8901} {Name:84c49a65b45392b MacAddress:82:3f:04:6f:12:9b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f5:ee:83:d7 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:19:80:44:73:98 Speed:0 Mtu:8901} {Name:c6f7b9afe84d1aa MacAddress:8a:a2:60:dd:eb:0e Speed:10000 Mtu:8901} {Name:e6d68cdb634edd3 MacAddress:62:e0:17:bc:aa:90 Speed:10000 Mtu:8901} {Name:e9340065a4ce117 MacAddress:8e:d4:cd:2d:1f:b7 Speed:10000 Mtu:8901} {Name:eebb12123ada46f MacAddress:4e:4e:b8:85:a9:8f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f5:ee:83:d7 Speed:0 Mtu:9001} {Name:fd3bc0faaada5f4 MacAddress:62:a4:85:97:f6:57 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f6:27:be:2c:27:0d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:27:6f:17:e9:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863340 5520 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863383 5520 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863673 5520 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863694 5520 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-153.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863813 5520 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863820 5520 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863830 5520 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863850 5520 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.863917 5520 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864077 5520 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864187 5520 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864246 5520 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864257 5520 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864267 5520 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864277 5520 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:16:15.864884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864285 5520 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:16:15.865701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864945 5520 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:16:15.865701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.864954 5520 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:16:15.865818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.865428 5520 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:16:15.868261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868220 5520 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:16:15.868261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868233 5520 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:16:15.868345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868335 5520 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 03:16:15.868535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868524 5520 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:16:15.868567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868553 5520 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:16:15.868653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868642 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:16:15.868686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868664 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:16:15.868686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868675 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:16:15.868686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868683 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868696 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868703 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868708 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868714 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868721 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868728 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868740 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:16:15.868772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868749 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:16:15.868990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868778 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:16:15.868990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.868787 5520 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:16:15.872418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.872403 5520 server.go:1263] "Started kubelet" Oct 31 03:16:15.874399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.873091 5520 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:16:15.874399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.872491 5520 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:16:15.874399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.874228 5520 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:16:15.873336 ip-10-0-141-153 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:16:15.875100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.875080 5520 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:16:15.876863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.876844 5520 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.878686 5520 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.878711 5520 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.878780 5520 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:10:06 +0000 UTC" deadline="2028-08-09 04:19:10.821872336 +0000 UTC" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.878809 5520 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.878810 5520 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15553h2m54.94306579s" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.878818 5520 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:15.878816 5520 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-153.us-east-2.compute.internal\" not found" Oct 31 03:16:15.879796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.878942 5520 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:16:15.880765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.880624 5520 factory.go:55] Registering systemd factory Oct 31 03:16:15.880765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.880643 5520 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881408 5520 factory.go:153] Registering CRI-O factory Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881427 5520 factory.go:223] Registration of the crio container factory successfully Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881479 5520 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881497 5520 factory.go:103] Registering Raw factory Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881514 5520 manager.go:1199] Started watching for new ooms in manager Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881813 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0577b36e-dd61-41bb-899d-c7af7b388696" volumeName="kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881848 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="196bf414-2c29-40a0-a957-f94cadaa7ff2" volumeName="kubernetes.io/configmap/196bf414-2c29-40a0-a957-f94cadaa7ff2-serviceca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881865 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="196bf414-2c29-40a0-a957-f94cadaa7ff2" volumeName="kubernetes.io/projected/196bf414-2c29-40a0-a957-f94cadaa7ff2-kube-api-access-vsljf" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881875 5520 manager.go:319] Starting recovery of all containers Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881879 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881893 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab0c10c-8503-43e7-8892-894c3c0650d4" volumeName="kubernetes.io/secret/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881907 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881920 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881933 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5deb6a1-16c9-4491-8e07-d8deafd8607f" volumeName="kubernetes.io/projected/e5deb6a1-16c9-4491-8e07-d8deafd8607f-kube-api-access-hbgtd" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881948 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453eddca-0701-4325-b5e2-12b4e3e78565" volumeName="kubernetes.io/configmap/453eddca-0701-4325-b5e2-12b4e3e78565-iptables-alerter-script" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881977 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453eddca-0701-4325-b5e2-12b4e3e78565" volumeName="kubernetes.io/projected/453eddca-0701-4325-b5e2-12b4e3e78565-kube-api-access-k4p9n" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881989 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab0c10c-8503-43e7-8892-894c3c0650d4" volumeName="kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-script-lib" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.881999 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85ec7f68-370c-4e14-a990-877b350fcdf8" volumeName="kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882012 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882032 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882046 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09f77c2-e722-4f39-b4c9-2fcca44268a9" volumeName="kubernetes.io/configmap/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882058 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f771de0f-1ffa-4de2-823d-40d0a95eed5c" volumeName="kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882071 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0577b36e-dd61-41bb-899d-c7af7b388696" volumeName="kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882083 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0577b36e-dd61-41bb-899d-c7af7b388696" volumeName="kubernetes.io/projected/0577b36e-dd61-41bb-899d-c7af7b388696-kube-api-access-ngg2k" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882093 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab0c10c-8503-43e7-8892-894c3c0650d4" volumeName="kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-env-overrides" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882104 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85ec7f68-370c-4e14-a990-877b350fcdf8" volumeName="kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882114 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07e2583-7e4e-49ef-8980-e76733289e46" volumeName="kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-etc-tuned" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882126 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882137 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882148 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882159 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882170 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df52d3e9-f738-4aaf-b8e4-de32943b6b03" volumeName="kubernetes.io/projected/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kube-api-access-dwvlv" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882180 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" volumeName="kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882193 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" volumeName="kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882233 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab0c10c-8503-43e7-8892-894c3c0650d4" volumeName="kubernetes.io/projected/5ab0c10c-8503-43e7-8892-894c3c0650d4-kube-api-access-72bmk" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882247 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882259 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c43876-cf34-43bb-b641-01e1b6a4b143" volumeName="kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882269 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4080580a-a99a-49e0-a255-06cb08581290" volumeName="kubernetes.io/secret/4080580a-a99a-49e0-a255-06cb08581290-cert" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882282 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85ec7f68-370c-4e14-a990-877b350fcdf8" volumeName="kubernetes.io/empty-dir/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-textfile" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882295 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882306 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5deb6a1-16c9-4491-8e07-d8deafd8607f" volumeName="kubernetes.io/configmap/e5deb6a1-16c9-4491-8e07-d8deafd8607f-config-volume" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882317 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5cd4aa-cca0-44d0-b314-05ed0788d784" volumeName="kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882328 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07e2583-7e4e-49ef-8980-e76733289e46" volumeName="kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-tmp" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882340 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0577b36e-dd61-41bb-899d-c7af7b388696" volumeName="kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882351 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3387800d-492d-4f65-b3d8-4bbccd906d03" volumeName="kubernetes.io/projected/3387800d-492d-4f65-b3d8-4bbccd906d03-kube-api-access-pnjxn" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882363 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f26e8e-228a-4b68-8851-c580c827e242" volumeName="kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882375 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85ec7f68-370c-4e14-a990-877b350fcdf8" volumeName="kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882387 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882399 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" volumeName="kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882410 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ab0c10c-8503-43e7-8892-894c3c0650d4" volumeName="kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882422 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b1ec6c2-6309-4e90-ae0b-b85aefc8c637" volumeName="kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882435 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f26e8e-228a-4b68-8851-c580c827e242" volumeName="kubernetes.io/configmap/75f26e8e-228a-4b68-8851-c580c827e242-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882446 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882457 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5deb6a1-16c9-4491-8e07-d8deafd8607f" volumeName="kubernetes.io/empty-dir/e5deb6a1-16c9-4491-8e07-d8deafd8607f-tmp-dir" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882469 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5cd4aa-cca0-44d0-b314-05ed0788d784" volumeName="kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882481 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09f77c2-e722-4f39-b4c9-2fcca44268a9" volumeName="kubernetes.io/secret/f09f77c2-e722-4f39-b4c9-2fcca44268a9-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882492 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85ec7f68-370c-4e14-a990-877b350fcdf8" volumeName="kubernetes.io/projected/85ec7f68-370c-4e14-a990-877b350fcdf8-kube-api-access-ffd55" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882504 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6a1947-49a2-44d8-9db3-5004938650e6" volumeName="kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-cni-binary-copy" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882515 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882527 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f454b995-5cf2-461d-b6f1-932bd4a45110" volumeName="kubernetes.io/configmap/f454b995-5cf2-461d-b6f1-932bd4a45110-konnectivity-ca" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882538 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4080580a-a99a-49e0-a255-06cb08581290" volumeName="kubernetes.io/projected/4080580a-a99a-49e0-a255-06cb08581290-kube-api-access-jzrs6" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882549 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73113784-937f-492c-9ac1-1a98c59d43e6" volumeName="kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882569 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6a1947-49a2-44d8-9db3-5004938650e6" volumeName="kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-daemon-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882580 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882591 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5cd4aa-cca0-44d0-b314-05ed0788d784" volumeName="kubernetes.io/projected/eb5cd4aa-cca0-44d0-b314-05ed0788d784-kube-api-access-tjkpf" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882602 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f771de0f-1ffa-4de2-823d-40d0a95eed5c" volumeName="kubernetes.io/projected/f771de0f-1ffa-4de2-823d-40d0a95eed5c-kube-api-access-8t2mm" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882614 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0577b36e-dd61-41bb-899d-c7af7b388696" volumeName="kubernetes.io/empty-dir/0577b36e-dd61-41bb-899d-c7af7b388696-volume-directive-shadow" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882624 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882634 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9c43876-cf34-43bb-b641-01e1b6a4b143" volumeName="kubernetes.io/projected/d9c43876-cf34-43bb-b641-01e1b6a4b143-kube-api-access-xvzf2" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882645 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5cd4aa-cca0-44d0-b314-05ed0788d784" volumeName="kubernetes.io/empty-dir/eb5cd4aa-cca0-44d0-b314-05ed0788d784-audit-log" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882656 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07e2583-7e4e-49ef-8980-e76733289e46" volumeName="kubernetes.io/projected/f07e2583-7e4e-49ef-8980-e76733289e46-kube-api-access-8ftdc" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882667 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09f77c2-e722-4f39-b4c9-2fcca44268a9" volumeName="kubernetes.io/empty-dir/f09f77c2-e722-4f39-b4c9-2fcca44268a9-data-volume" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882677 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f771de0f-1ffa-4de2-823d-40d0a95eed5c" volumeName="kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882688 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" volumeName="kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882700 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3387800d-492d-4f65-b3d8-4bbccd906d03" volumeName="kubernetes.io/empty-dir/3387800d-492d-4f65-b3d8-4bbccd906d03-tmp-dir" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882712 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f26e8e-228a-4b68-8851-c580c827e242" volumeName="kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882723 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85ec7f68-370c-4e14-a990-877b350fcdf8" volumeName="kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-tls" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882734 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882746 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5deb6a1-16c9-4491-8e07-d8deafd8607f" volumeName="kubernetes.io/secret/e5deb6a1-16c9-4491-8e07-d8deafd8607f-metrics-tls" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882757 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5cd4aa-cca0-44d0-b314-05ed0788d784" volumeName="kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882769 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882780 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" volumeName="kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882793 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6a1947-49a2-44d8-9db3-5004938650e6" volumeName="kubernetes.io/projected/8e6a1947-49a2-44d8-9db3-5004938650e6-kube-api-access-sk9db" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882804 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882816 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a44a09c8-5e56-478e-b27f-eca60624a37b" volumeName="kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882826 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f09f77c2-e722-4f39-b4c9-2fcca44268a9" volumeName="kubernetes.io/projected/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-api-access-kc8z5" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882837 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f771de0f-1ffa-4de2-823d-40d0a95eed5c" volumeName="kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-binary-copy" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882850 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" volumeName="kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882862 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" volumeName="kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882876 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f454b995-5cf2-461d-b6f1-932bd4a45110" volumeName="kubernetes.io/secret/f454b995-5cf2-461d-b6f1-932bd4a45110-agent-certs" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882888 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0577b36e-dd61-41bb-899d-c7af7b388696" volumeName="kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882899 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f26e8e-228a-4b68-8851-c580c827e242" volumeName="kubernetes.io/projected/75f26e8e-228a-4b68-8851-c580c827e242-kube-api-access-srtz9" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882910 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954e8b0e-c426-4cee-80d6-e296726215d4" volumeName="kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882921 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5cd4aa-cca0-44d0-b314-05ed0788d784" volumeName="kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-client-ca-bundle" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882931 5520 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb5cd4aa-cca0-44d0-b314-05ed0788d784" volumeName="kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882943 5520 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:16:15.883310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.882951 5520 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:16:15.887971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.885667 5520 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:15.887971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.885893 5520 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:15.888045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.887997 5520 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:15.888901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.888875 5520 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 03:16:15.907455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.907062 5520 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:16:15.908810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.908759 5520 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:16:15.908810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.908786 5520 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:16:15.908810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.908812 5520 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:16:15.909019 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:15.908854 5520 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 03:16:15.912959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.912934 5520 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:15.923952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.923929 5520 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" exitCode=0 Oct 31 03:16:15.930546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.930517 5520 generic.go:356] "Generic (PLEG): container finished" podID="b6a0fa3bb14222e5d3d01c4ed0d8f38f" containerID="97382951ed0825677e6b5466d67f742c206dfec8d048046d1559607a9423a74d" exitCode=0 Oct 31 03:16:15.948014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.947990 5520 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="70c2bff2c716a2d9b94078b1ad43feefba802bdcbdd760cdd95978d05d001a3e" exitCode=0 Oct 31 03:16:15.948014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.948012 5520 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="0282c04de41c04e57c8eb4d4d09bb07f39bd2464d3b89bcbe83ea4ade0497aee" exitCode=0 Oct 31 03:16:15.948148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.948019 5520 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="810f4686f7984e446e9472ea40d1ecbd573c56c6a71980da65675f8d5b621e57" exitCode=0 Oct 31 03:16:15.948148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.948024 5520 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="48fbebf18ffb7d0d342d7923b0658462078ab4ba87dd41db81f3ae7e4d87d0dc" exitCode=0 Oct 31 03:16:15.948148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.948030 5520 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="8b11705232151f4fc803cd9bf5c432dd4fa1ffea5fa6f0225eeea84788bd0a94" exitCode=0 Oct 31 03:16:15.948148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.948035 5520 generic.go:356] "Generic (PLEG): container finished" podID="f771de0f-1ffa-4de2-823d-40d0a95eed5c" containerID="e714158845edecc29e563084c43fd3127f86775c95f4e582d59d49723720c0c3" exitCode=0 Oct 31 03:16:15.958375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.958352 5520 generic.go:356] "Generic (PLEG): container finished" podID="85ec7f68-370c-4e14-a990-877b350fcdf8" containerID="08ef0113cc7f877c182ff7fbfd7cda54019e6d6bec02247fe681a95d723661a5" exitCode=0 Oct 31 03:16:15.986774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:15.986753 5520 manager.go:324] Recovery completed Oct 31 03:16:16.008684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008609 5520 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:16:16.008684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008636 5520 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:16:16.008684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008665 5520 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:16:16.008887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008864 5520 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 03:16:16.008932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008879 5520 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 03:16:16.008932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008903 5520 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 03:16:16.008932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008911 5520 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 03:16:16.008932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008918 5520 policy_none.go:50] "Start" Oct 31 03:16:16.009108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008937 5520 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:16:16.009108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.008949 5520 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:16:16.009188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.009165 5520 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 03:16:16.009188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.009176 5520 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 03:16:16.009188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.009184 5520 policy_none.go:44] "Start" Oct 31 03:16:16.009372 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:16.009344 5520 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 03:16:16.014287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.014260 5520 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:16:16.016943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.016927 5520 manager.go:341] "Starting Device Plugin manager" Oct 31 03:16:16.017641 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:16.016950 5520 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:16:16.017641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.017641 5520 server.go:87] "Starting device plugin registration server" Oct 31 03:16:16.017973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.017957 5520 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:16:16.018057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.017972 5520 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:16:16.018057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.018038 5520 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:16:16.018166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.018133 5520 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:16:16.018254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.018181 5520 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:16:16.018254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.018188 5520 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:16:16.018370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.018251 5520 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:16:16.018164417Z","UUID":"8b059422-2175-40b1-926a-03634d1f2563","Handler":null,"Name":"","Endpoint":""} Oct 31 03:16:16.019681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.019664 5520 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:16:16.019778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.019692 5520 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:16:16.020191 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:16.019980 5520 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:16:16.118855 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.118825 5520 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:16:16.121026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.121002 5520 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:16:16.121147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.121033 5520 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:16:16.121147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.121044 5520 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:16:16.121147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.121086 5520 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.132613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.132591 5520 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.132668 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.132659 5520 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.210463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.210400 5520 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal"] Oct 31 03:16:16.211036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.210994 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"b6a0fa3bb14222e5d3d01c4ed0d8f38f","Type":"ContainerStarted","Data":"ab304681b02447ce6b723f4cf3792013501bda867b9ace1bbae3c02103f14783"} Oct 31 03:16:16.211129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.211048 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"b6a0fa3bb14222e5d3d01c4ed0d8f38f","Type":"ContainerDied","Data":"97382951ed0825677e6b5466d67f742c206dfec8d048046d1559607a9423a74d"} Oct 31 03:16:16.211129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.211066 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"b6a0fa3bb14222e5d3d01c4ed0d8f38f","Type":"ContainerStarted","Data":"41f4b70e7591d13486748ce804c85935fd06b57f90db19f387e4cceffb1a1200"} Oct 31 03:16:16.211129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.211090 5520 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.211292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.211145 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"057a5e637a257f84120786f81c645c34","Type":"ContainerStarted","Data":"40e867ec31b9920e828c419c6aaf2fbd87b2330fc196d41fbcfeae3621d8876f"} Oct 31 03:16:16.211292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.211168 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" event={"ID":"057a5e637a257f84120786f81c645c34","Type":"ContainerStarted","Data":"0c89fa1fc434421100bef497f3087d32115a4ebfeec8f000a0dbd03d25b4bd5a"} Oct 31 03:16:16.211356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.211328 5520 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.222822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.222803 5520 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:16.222872 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:16.222849 5520 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.223351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.223338 5520 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:16.223393 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:16.223370 5520 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.290749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.290666 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.290873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.290753 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.290873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.290837 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a5e637a257f84120786f81c645c34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"057a5e637a257f84120786f81c645c34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.391338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.391304 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.391466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.391340 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.391466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.391361 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a5e637a257f84120786f81c645c34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"057a5e637a257f84120786f81c645c34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.391466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.391405 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/057a5e637a257f84120786f81c645c34-config\") pod \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"057a5e637a257f84120786f81c645c34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.391466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.391417 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.391466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.391424 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a0fa3bb14222e5d3d01c4ed0d8f38f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" (UID: \"b6a0fa3bb14222e5d3d01c4ed0d8f38f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.865070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.865026 5520 apiserver.go:52] "Watching apiserver" Oct 31 03:16:16.874763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.874736 5520 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:16:16.875347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.875326 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j","openshift-multus/network-metrics-daemon-flc2b","openshift-network-operator/iptables-alerter-ghj2p","kube-system/global-pull-secret-syncer-9cczw","openshift-console/console-66fc4df878-chnw2","openshift-image-registry/node-ca-xwkgw","openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz","openshift-network-diagnostics/network-check-target-dlvjw","openshift-monitoring/metrics-server-5d97946f89-wfdp6","openshift-cluster-node-tuning-operator/tuned-gfmjc","openshift-ingress-canary/ingress-canary-vbv64","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal","openshift-console/console-7f4c575df7-h42hr","openshift-monitoring/node-exporter-gsvct","openshift-ovn-kubernetes/ovnkube-node-8z9f4","openshift-multus/multus-additional-cni-plugins-rrqw6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6","openshift-dns/node-resolver-dbm7r","kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal","openshift-dns/dns-default-76rrr","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-gqnj2","kube-system/konnectivity-agent-t5fpc","openshift-insights/insights-runtime-extractor-s8l89"] Oct 31 03:16:16.890817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.890789 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b55zh\"" Oct 31 03:16:16.890962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.890821 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmjq8\"" Oct 31 03:16:16.890962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.890826 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:16:16.890962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.890885 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6lm2\"" Oct 31 03:16:16.890962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.890888 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nklpg\"" Oct 31 03:16:16.892909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.892886 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.892977 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.892919 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:16:16.893038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893023 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-lib-modules\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.893072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893046 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-host\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.893135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893067 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9db\" (UniqueName: \"kubernetes.io/projected/8e6a1947-49a2-44d8-9db3-5004938650e6-kube-api-access-sk9db\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.893135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893088 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-netns\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.893135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893104 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:16:16.893290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893146 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-var-lib-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.893290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893227 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-sys\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.893383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893323 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-var-lib-kubelet\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.893383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893360 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-etc-tuned\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.893466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893397 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-socket-dir-parent\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.893466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893413 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 03:16:16.893466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893421 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:16:16.893466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893443 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-etc-tuned\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.893576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893498 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.893576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893561 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-registration-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.893633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893608 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/453eddca-0701-4325-b5e2-12b4e3e78565-iptables-alerter-script\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:16:16.893665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893652 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-conf\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.893724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893710 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-device-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.893760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893747 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cnibin\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.893792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893772 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196bf414-2c29-40a0-a957-f94cadaa7ff2-host\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:16:16.893823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893804 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-etc-kubernetes\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.893852 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893840 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvzf2\" (UniqueName: \"kubernetes.io/projected/d9c43876-cf34-43bb-b641-01e1b6a4b143-kube-api-access-xvzf2\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:16:16.893893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893881 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3387800d-492d-4f65-b3d8-4bbccd906d03-hosts-file\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:16:16.893927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893893 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:16:16.893961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893942 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:16:16.894011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.893993 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.894058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894030 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-daemon-config\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.894096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894060 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjxn\" (UniqueName: \"kubernetes.io/projected/3387800d-492d-4f65-b3d8-4bbccd906d03-kube-api-access-pnjxn\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:16:16.894159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894143 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-bin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.894229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894145 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\"" Oct 31 03:16:16.894229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894175 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-kubelet\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.894229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894177 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 03:16:16.894229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894217 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:16:16.894405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894240 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.894405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894263 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.894405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894291 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsljf\" (UniqueName: \"kubernetes.io/projected/196bf414-2c29-40a0-a957-f94cadaa7ff2-kube-api-access-vsljf\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:16:16.894405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894321 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5s7vx\"" Oct 31 03:16:16.894405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894340 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-k8s-cni-cncf-io\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.894405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894370 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qg7r7\"" Oct 31 03:16:16.894405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894386 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p9n\" (UniqueName: \"kubernetes.io/projected/453eddca-0701-4325-b5e2-12b4e3e78565-kube-api-access-k4p9n\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894421 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894454 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-systemd\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894463 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894482 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-tmp\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894492 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894523 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894539 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftdc\" (UniqueName: \"kubernetes.io/projected/f07e2583-7e4e-49ef-8980-e76733289e46-kube-api-access-8ftdc\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894571 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e2583-7e4e-49ef-8980-e76733289e46-tmp\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894581 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894630 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-slash\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.894661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894647 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-config\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894698 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-script-lib\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894735 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysconfig\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894761 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f454b995-5cf2-461d-b6f1-932bd4a45110-konnectivity-ca\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894776 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894798 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-etc-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894820 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-node-log\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894842 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-socket-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894879 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894924 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-multus\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894938 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f454b995-5cf2-461d-b6f1-932bd4a45110-konnectivity-ca\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894975 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-hostroot\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.895013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.894980 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895034 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-system-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895058 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895072 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895116 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-log-socket\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895143 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-kubernetes\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895160 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-conf-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895179 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-netd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895313 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895321 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f454b995-5cf2-461d-b6f1-932bd4a45110-agent-certs\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895369 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.895540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895457 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-cnibin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895546 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f454b995-5cf2-461d-b6f1-932bd4a45110-agent-certs\") pod \"konnectivity-agent-t5fpc\" (UID: \"f454b995-5cf2-461d-b6f1-932bd4a45110\") " pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895554 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-kubelet\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895584 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-systemd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895609 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-bin\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895634 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895662 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvlv\" (UniqueName: \"kubernetes.io/projected/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kube-api-access-dwvlv\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895688 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2mm\" (UniqueName: \"kubernetes.io/projected/f771de0f-1ffa-4de2-823d-40d0a95eed5c-kube-api-access-8t2mm\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895717 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-modprobe-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895743 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-kubelet-config\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895775 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/196bf414-2c29-40a0-a957-f94cadaa7ff2-serviceca\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895886 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895904 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/453eddca-0701-4325-b5e2-12b4e3e78565-host-slash\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895929 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-systemd-units\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895959 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bmk\" (UniqueName: \"kubernetes.io/projected/5ab0c10c-8503-43e7-8892-894c3c0650d4-kube-api-access-72bmk\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.895992 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3387800d-492d-4f65-b3d8-4bbccd906d03-tmp-dir\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:16:16.896030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896009 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-cni-binary-copy\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896046 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-env-overrides\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896175 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-cni-binary-copy\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896191 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-run\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896227 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3387800d-492d-4f65-b3d8-4bbccd906d03-tmp-dir\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896235 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-sys-fs\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896251 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-system-cni-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896266 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-dbus\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896280 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-multus-certs\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896294 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-netns\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896335 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-ovn\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896392 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovn-node-metrics-cert\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.896558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896426 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:16:16.896875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.896619 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9c43876-cf34-43bb-b641-01e1b6a4b143-metrics-certs\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:16:16.899225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.899183 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rtvr4\"" Oct 31 03:16:16.899759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.899738 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:16:16.899868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.899806 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fn5cc\"" Oct 31 03:16:16.899868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.899809 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkjgm\"" Oct 31 03:16:16.899868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.899807 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 03:16:16.900031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.899835 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9h5x8\"" Oct 31 03:16:16.900031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.899850 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:16:16.900123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900051 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:16:16.900123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900054 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\"" Oct 31 03:16:16.900123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900098 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dct2q\"" Oct 31 03:16:16.900332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900257 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:16:16.900332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900281 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\"" Oct 31 03:16:16.900332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900295 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p2pq6\"" Oct 31 03:16:16.900332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900286 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zf8rk\"" Oct 31 03:16:16.900332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900298 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 03:16:16.900332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900327 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 03:16:16.900332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900328 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:16:16.900650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900355 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dk5h8\"" Oct 31 03:16:16.900650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.900396 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ztddf\"" Oct 31 03:16:16.904712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.904695 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-daemon-config\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.904871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.904857 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f771de0f-1ffa-4de2-823d-40d0a95eed5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.905970 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.905947 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/73113784-937f-492c-9ac1-1a98c59d43e6-original-pull-secret\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.906977 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.906962 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovn-node-metrics-cert\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.916653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.916632 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2mm\" (UniqueName: \"kubernetes.io/projected/f771de0f-1ffa-4de2-823d-40d0a95eed5c-kube-api-access-8t2mm\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.928300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.928278 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9db\" (UniqueName: \"kubernetes.io/projected/8e6a1947-49a2-44d8-9db3-5004938650e6-kube-api-access-sk9db\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.929089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.929073 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvzf2\" (UniqueName: \"kubernetes.io/projected/d9c43876-cf34-43bb-b641-01e1b6a4b143-kube-api-access-xvzf2\") pod \"network-metrics-daemon-flc2b\" (UID: \"d9c43876-cf34-43bb-b641-01e1b6a4b143\") " pod="openshift-multus/network-metrics-daemon-flc2b" Oct 31 03:16:16.966530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.966505 5520 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.966675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.966619 5520 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.980492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.980459 5520 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:16:16.986277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986257 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 03:16:16.986404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986275 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 03:16:16.986521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986503 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 03:16:16.986521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986514 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.986650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986552 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.986650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986578 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 03:16:16.986798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986774 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.986934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986817 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:16:16.986934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986834 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.986934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986836 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 03:16:16.986934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986835 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.986934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986863 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.986934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.986921 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.987301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987003 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.987301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987048 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 03:16:16.987301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987071 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.987301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987240 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 03:16:16.987301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987266 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987361 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987365 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987365 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987369 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987394 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987369 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987485 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987512 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.987533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987533 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987587 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987594 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987603 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987625 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987766 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987790 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987818 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987827 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987915 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987917 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987926 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987960 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987968 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987986 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988005 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nz4s9\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988013 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.987970 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988008 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7h7nd\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988045 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a61rlu4tvuhve\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988047 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:16:16.988071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988052 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.988879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988089 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:16:16.988879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988141 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:16:16.988879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988291 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rpnsw\"" Oct 31 03:16:16.988879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988341 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:16:16.988879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988353 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 03:16:16.988879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988537 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsljf\" (UniqueName: \"kubernetes.io/projected/196bf414-2c29-40a0-a957-f94cadaa7ff2-kube-api-access-vsljf\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:16:16.989101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.988995 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftdc\" (UniqueName: \"kubernetes.io/projected/f07e2583-7e4e-49ef-8980-e76733289e46-kube-api-access-8ftdc\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.989251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.989191 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 03:16:16.993188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.991125 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 03:16:16.993188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.991746 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfm56\" (UniqueName: \"kubernetes.io/projected/6b1ec6c2-6309-4e90-ae0b-b85aefc8c637-kube-api-access-kfm56\") pod \"network-check-target-dlvjw\" (UID: \"6b1ec6c2-6309-4e90-ae0b-b85aefc8c637\") " pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:16:16.994168 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.994152 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/453eddca-0701-4325-b5e2-12b4e3e78565-iptables-alerter-script\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:16:16.995089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.995071 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-config\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.995195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.995178 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-ovnkube-script-lib\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.995264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.995236 5520 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:16.995295 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:16.995271 5520 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.996283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996264 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ab0c10c-8503-43e7-8892-894c3c0650d4-env-overrides\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.996419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996298 5520 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:16.996419 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:16.996349 5520 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-153.us-east-2.compute.internal" Oct 31 03:16:16.996529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996511 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bmk\" (UniqueName: \"kubernetes.io/projected/5ab0c10c-8503-43e7-8892-894c3c0650d4-kube-api-access-72bmk\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.996613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996593 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/196bf414-2c29-40a0-a957-f94cadaa7ff2-serviceca\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:16:16.996700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996685 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjxn\" (UniqueName: \"kubernetes.io/projected/3387800d-492d-4f65-b3d8-4bbccd906d03-kube-api-access-pnjxn\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:16:16.996756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996741 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvlv\" (UniqueName: \"kubernetes.io/projected/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kube-api-access-dwvlv\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.996756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996752 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-bin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.996815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996773 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.996815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996795 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-k8s-cni-cncf-io\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.996815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996804 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-bin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.996895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996846 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-systemd\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.996895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996810 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-systemd\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.996895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996860 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.996895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996877 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-k8s-cni-cncf-io\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.996895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996884 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkpf\" (UniqueName: \"kubernetes.io/projected/eb5cd4aa-cca0-44d0-b314-05ed0788d784-kube-api-access-tjkpf\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996907 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996924 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-slash\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996945 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-metrics-server-audit-profiles\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996966 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-slash\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996971 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.996994 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997009 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtz9\" (UniqueName: \"kubernetes.io/projected/75f26e8e-228a-4b68-8851-c580c827e242-kube-api-access-srtz9\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:16.997032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997025 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-root\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997045 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-socket-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997066 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-accelerators-collector-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997078 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f09f77c2-e722-4f39-b4c9-2fcca44268a9-crio-socket\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997219 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-socket-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997231 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-metrics-server-audit-profiles\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997243 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997225 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8z5\" (UniqueName: \"kubernetes.io/projected/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-api-access-kc8z5\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997276 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997293 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-multus\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997316 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-hostroot\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997331 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997332 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997352 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-log-socket\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997359 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-cni-multus\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997387 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-hostroot\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997388 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-kubernetes\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.997412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997413 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997393 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-log-socket\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997430 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-client-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997430 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-kubernetes\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997460 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-conf-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997486 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-conf-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997471 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997500 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-netd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997531 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-netd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997546 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997577 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997599 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-client-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997601 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997639 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997665 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997692 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-sys\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997717 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-bin\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997744 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997755 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997773 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997778 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997785 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997804 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997808 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-cni-bin\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997832 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997841 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997854 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997858 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f26e8e-228a-4b68-8851-c580c827e242-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997854 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997882 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997898 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-kubelet-config\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997932 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/85ec7f68-370c-4e14-a990-877b350fcdf8-kube-api-access-ffd55\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997956 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-run\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997977 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-sys-fs\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.997998 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-system-cni-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998021 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-dbus\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998023 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f26e8e-228a-4b68-8851-c580c827e242-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998029 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998035 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998020 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998059 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998091 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998128 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-sys-fs\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998164 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-run\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.998189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998219 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-kubelet-config\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998305 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p9n\" (UniqueName: \"kubernetes.io/projected/453eddca-0701-4325-b5e2-12b4e3e78565-kube-api-access-k4p9n\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998346 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-system-cni-dir\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998382 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/73113784-937f-492c-9ac1-1a98c59d43e6-dbus\") pod \"global-pull-secret-syncer-9cczw\" (UID: \"73113784-937f-492c-9ac1-1a98c59d43e6\") " pod="kube-system/global-pull-secret-syncer-9cczw" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998420 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998439 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-netns\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998454 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-ovn\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998469 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998476 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-netns\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998502 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-ovn\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998523 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-tls\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998539 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-var-lib-kubelet\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998558 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4080580a-a99a-49e0-a255-06cb08581290-cert\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998578 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998583 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998620 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998627 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-var-lib-kubelet\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998644 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998668 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998690 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-device-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998700 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998713 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196bf414-2c29-40a0-a957-f94cadaa7ff2-host\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998718 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998739 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998741 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998753 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-device-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998764 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3387800d-492d-4f65-b3d8-4bbccd906d03-hosts-file\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998792 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998799 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3387800d-492d-4f65-b3d8-4bbccd906d03-hosts-file\") pod \"node-resolver-dbm7r\" (UID: \"3387800d-492d-4f65-b3d8-4bbccd906d03\") " pod="openshift-dns/node-resolver-dbm7r" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998806 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4080580a-a99a-49e0-a255-06cb08581290-cert\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998816 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f09f77c2-e722-4f39-b4c9-2fcca44268a9-data-volume\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998767 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196bf414-2c29-40a0-a957-f94cadaa7ff2-host\") pod \"node-ca-xwkgw\" (UID: \"196bf414-2c29-40a0-a957-f94cadaa7ff2\") " pod="openshift-image-registry/node-ca-xwkgw" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998763 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-tls\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998840 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998845 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998855 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998872 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998903 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998913 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f09f77c2-e722-4f39-b4c9-2fcca44268a9-data-volume\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998919 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.998926 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-kubelet\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999022 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-kubelet\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999022 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999055 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999093 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999108 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999132 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5deb6a1-16c9-4491-8e07-d8deafd8607f-metrics-tls\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999163 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkfz\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999172 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0577b36e-dd61-41bb-899d-c7af7b388696-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999189 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999242 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999248 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999192 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999266 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysconfig\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999294 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999302 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysconfig\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999318 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-etc-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999338 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e5deb6a1-16c9-4491-8e07-d8deafd8607f-metrics-tls\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999364 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-etc-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999390 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-node-log\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999408 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999440 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0577b36e-dd61-41bb-899d-c7af7b388696-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999453 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-node-log\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999465 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vnm\" (UniqueName: \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999469 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75f26e8e-228a-4b68-8851-c580c827e242-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999489 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-etc-selinux\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999498 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0577b36e-dd61-41bb-899d-c7af7b388696-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999491 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-system-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999534 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999559 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-metrics-client-ca\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999565 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999534 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-system-cni-dir\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999583 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999614 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-cnibin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999638 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-kubelet\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999661 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-systemd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999685 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999694 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-cnibin\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999731 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-var-lib-kubelet\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999739 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-run-systemd\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999749 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85ec7f68-370c-4e14-a990-877b350fcdf8-metrics-client-ca\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999765 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:16.999720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999778 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999811 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999785 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5deb6a1-16c9-4491-8e07-d8deafd8607f-config-volume\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999850 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-modprobe-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999886 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/453eddca-0701-4325-b5e2-12b4e3e78565-host-slash\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999910 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-systemd-units\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999936 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/453eddca-0701-4325-b5e2-12b4e3e78565-host-slash\") pod \"iptables-alerter-ghj2p\" (UID: \"453eddca-0701-4325-b5e2-12b4e3e78565\") " pod="openshift-network-operator/iptables-alerter-ghj2p" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999942 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb5cd4aa-cca0-44d0-b314-05ed0788d784-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999957 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999965 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-systemd-units\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999969 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-modprobe-d\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999958 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e5deb6a1-16c9-4491-8e07-d8deafd8607f-config-volume\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:16.999981 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgtd\" (UniqueName: \"kubernetes.io/projected/e5deb6a1-16c9-4491-8e07-d8deafd8607f-kube-api-access-hbgtd\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000007 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-wtmp\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000027 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000072 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f09f77c2-e722-4f39-b4c9-2fcca44268a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000096 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000127 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-multus-certs\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000150 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-host-run-multus-certs\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000158 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000189 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-lib-modules\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000217 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0577b36e-dd61-41bb-899d-c7af7b388696-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000230 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-host\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000261 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngg2k\" (UniqueName: \"kubernetes.io/projected/0577b36e-dd61-41bb-899d-c7af7b388696-kube-api-access-ngg2k\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000289 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxcr\" (UniqueName: \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000265 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000304 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f09f77c2-e722-4f39-b4c9-2fcca44268a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000292 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-host\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000321 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrs6\" (UniqueName: \"kubernetes.io/projected/4080580a-a99a-49e0-a255-06cb08581290-kube-api-access-jzrs6\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000322 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-lib-modules\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000353 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-netns\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000378 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-var-lib-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000400 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-sys\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000405 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-host-run-netns\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000426 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000432 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-sys\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000420 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ab0c10c-8503-43e7-8892-894c3c0650d4-var-lib-openvswitch\") pod \"ovnkube-node-8z9f4\" (UID: \"5ab0c10c-8503-43e7-8892-894c3c0650d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000456 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-socket-dir-parent\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000473 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-registration-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000496 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-tls\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000523 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-textfile\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000530 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-multus-socket-dir-parent\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000547 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-conf\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000571 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df52d3e9-f738-4aaf-b8e4-de32943b6b03-registration-dir\") pod \"aws-ebs-csi-driver-node-dn2d6\" (UID: \"df52d3e9-f738-4aaf-b8e4-de32943b6b03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dn2d6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000571 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cnibin\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000599 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-client-certs\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000609 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-textfile\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000622 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f771de0f-1ffa-4de2-823d-40d0a95eed5c-cnibin\") pod \"multus-additional-cni-plugins-rrqw6\" (UID: \"f771de0f-1ffa-4de2-823d-40d0a95eed5c\") " pod="openshift-multus/multus-additional-cni-plugins-rrqw6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000626 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000679 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000689 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5deb6a1-16c9-4491-8e07-d8deafd8607f-tmp-dir\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000712 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e2583-7e4e-49ef-8980-e76733289e46-etc-sysctl-conf\") pod \"tuned-gfmjc\" (UID: \"f07e2583-7e4e-49ef-8980-e76733289e46\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmjc" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000737 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-etc-kubernetes\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000752 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-tls\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000759 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5deb6a1-16c9-4491-8e07-d8deafd8607f-tmp-dir\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000791 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e6a1947-49a2-44d8-9db3-5004938650e6-etc-kubernetes\") pod \"multus-gqnj2\" (UID: \"8e6a1947-49a2-44d8-9db3-5004938650e6\") " pod="openshift-multus/multus-gqnj2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000807 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb5cd4aa-cca0-44d0-b314-05ed0788d784-audit-log\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000812 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000835 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000857 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb5cd4aa-cca0-44d0-b314-05ed0788d784-secret-metrics-server-client-certs\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000861 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.000901 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb5cd4aa-cca0-44d0-b314-05ed0788d784-audit-log\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.001012 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:17.002385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.001057 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:17.004179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.003824 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkpf\" (UniqueName: \"kubernetes.io/projected/eb5cd4aa-cca0-44d0-b314-05ed0788d784-kube-api-access-tjkpf\") pod \"metrics-server-5d97946f89-wfdp6\" (UID: \"eb5cd4aa-cca0-44d0-b314-05ed0788d784\") " pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:17.008684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008498 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtz9\" (UniqueName: \"kubernetes.io/projected/75f26e8e-228a-4b68-8851-c580c827e242-kube-api-access-srtz9\") pod \"openshift-state-metrics-7bd7489957-6lkfz\" (UID: \"75f26e8e-228a-4b68-8851-c580c827e242\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6lkfz" Oct 31 03:16:17.008684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008632 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffd55\" (UniqueName: \"kubernetes.io/projected/85ec7f68-370c-4e14-a990-877b350fcdf8-kube-api-access-ffd55\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.008856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008714 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8z5\" (UniqueName: \"kubernetes.io/projected/f09f77c2-e722-4f39-b4c9-2fcca44268a9-kube-api-access-kc8z5\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:17.008913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008882 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngg2k\" (UniqueName: \"kubernetes.io/projected/0577b36e-dd61-41bb-899d-c7af7b388696-kube-api-access-ngg2k\") pod \"kube-state-metrics-5dc8d7d89b-r4j9j\" (UID: \"0577b36e-dd61-41bb-899d-c7af7b388696\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-r4j9j" Oct 31 03:16:17.008991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008928 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkfz\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\") pod \"alertmanager-main-0\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:17.008991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008951 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vnm\" (UniqueName: \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\") pod \"console-66fc4df878-chnw2\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:17.008991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008933 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrs6\" (UniqueName: \"kubernetes.io/projected/4080580a-a99a-49e0-a255-06cb08581290-kube-api-access-jzrs6\") pod \"ingress-canary-vbv64\" (UID: \"4080580a-a99a-49e0-a255-06cb08581290\") " pod="openshift-ingress-canary/ingress-canary-vbv64" Oct 31 03:16:17.009141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.008994 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgtd\" (UniqueName: \"kubernetes.io/projected/e5deb6a1-16c9-4491-8e07-d8deafd8607f-kube-api-access-hbgtd\") pod \"dns-default-76rrr\" (UID: \"e5deb6a1-16c9-4491-8e07-d8deafd8607f\") " pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:17.009249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.009228 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxcr\" (UniqueName: \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\") pod \"console-7f4c575df7-h42hr\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:17.101536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101500 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-root\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.101536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101538 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f09f77c2-e722-4f39-b4c9-2fcca44268a9-crio-socket\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:17.101783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101568 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-sys\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.101783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101611 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-root\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.101783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101626 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-wtmp\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.101783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101632 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f09f77c2-e722-4f39-b4c9-2fcca44268a9-crio-socket\") pod \"insights-runtime-extractor-s8l89\" (UID: \"f09f77c2-e722-4f39-b4c9-2fcca44268a9\") " pod="openshift-insights/insights-runtime-extractor-s8l89" Oct 31 03:16:17.101783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101689 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-sys\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:17.101783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:17.101747 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85ec7f68-370c-4e14-a990-877b350fcdf8-node-exporter-wtmp\") pod \"node-exporter-gsvct\" (UID: \"85ec7f68-370c-4e14-a990-877b350fcdf8\") " pod="openshift-monitoring/node-exporter-gsvct" Oct 31 03:16:18.217048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:18.217010 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:16:18.834014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:18.833988 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:18.942242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:18.942192 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-76rrr" Oct 31 03:16:21.181600 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:21.181564 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:16:23.064881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:23.064845 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d97946f89-wfdp6" Oct 31 03:16:26.645308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:26.645230 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:26.645824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:26.645470 5520 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:16:27.525266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.525236 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:16:27.525553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.525507 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="alertmanager" containerID="cri-o://a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" gracePeriod=120 Oct 31 03:16:27.525620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.525583 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-metric" containerID="cri-o://0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" gracePeriod=120 Oct 31 03:16:27.525674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.525620 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-web" containerID="cri-o://f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" gracePeriod=120 Oct 31 03:16:27.525674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.525624 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="prom-label-proxy" containerID="cri-o://6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" gracePeriod=120 Oct 31 03:16:27.525757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.525670 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="config-reloader" containerID="cri-o://7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" gracePeriod=120 Oct 31 03:16:27.525757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.525710 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy" containerID="cri-o://12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" gracePeriod=120 Oct 31 03:16:27.828999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.828967 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:27.874891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.874863 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.874899 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.874927 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.874957 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875010 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875046 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875086 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875134 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875165 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875194 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875255 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875297 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875346 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"954e8b0e-c426-4cee-80d6-e296726215d4\" (UID: \"954e8b0e-c426-4cee-80d6-e296726215d4\") " Oct 31 03:16:27.875704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875608 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:27.875845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.875820 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:27.876483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.876453 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:16:27.879274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.878918 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:16:27.879274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.878933 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:27.879441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.879325 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:27.881246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.881220 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:27.881343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.881259 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:27.881446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.881407 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "kube-api-access-hdkfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:27.881532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.881437 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:27.881759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.881729 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:27.884690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.884665 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:27.890651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.890626 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config" pod "954e8b0e-c426-4cee-80d6-e296726215d4" (UID: "954e8b0e-c426-4cee-80d6-e296726215d4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:27.976002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.975964 5520 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-cluster-tls-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.975997 5520 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-tls-assets\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976008 5520 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-main-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976018 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkfz\" (UniqueName: \"kubernetes.io/projected/954e8b0e-c426-4cee-80d6-e296726215d4-kube-api-access-hdkfz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976027 5520 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976037 5520 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976047 5520 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-web-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976055 5520 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-main-db\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976065 5520 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976073 5520 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976081 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/954e8b0e-c426-4cee-80d6-e296726215d4-metrics-client-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976090 5520 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/954e8b0e-c426-4cee-80d6-e296726215d4-config-out\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:27.976273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:27.976098 5520 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/954e8b0e-c426-4cee-80d6-e296726215d4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:28.005708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005678 5520 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" exitCode=0 Oct 31 03:16:28.005708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005701 5520 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" exitCode=0 Oct 31 03:16:28.005708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005708 5520 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" exitCode=0 Oct 31 03:16:28.005708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005716 5520 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" exitCode=0 Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005727 5520 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" exitCode=0 Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005739 5520 generic.go:356] "Generic (PLEG): container finished" podID="954e8b0e-c426-4cee-80d6-e296726215d4" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" exitCode=0 Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005757 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005772 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005883 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005929 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005944 5520 scope.go:122] "RemoveContainer" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.005971 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.006014 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} Oct 31 03:16:28.006060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.006055 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} Oct 31 03:16:28.006504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.006093 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"954e8b0e-c426-4cee-80d6-e296726215d4","Type":"ContainerDied","Data":"3da2874dc6071ab92d8d908442e06885a816efdb43a6a097209b185d1a2fe5ca"} Oct 31 03:16:28.020612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.020594 5520 scope.go:122] "RemoveContainer" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.026050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.026025 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:16:28.028761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.028736 5520 scope.go:122] "RemoveContainer" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.030238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.030218 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:16:28.036157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.036138 5520 scope.go:122] "RemoveContainer" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.042948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.042935 5520 scope.go:122] "RemoveContainer" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.049937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.049916 5520 scope.go:122] "RemoveContainer" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.057302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.057285 5520 scope.go:122] "RemoveContainer" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.058433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058413 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:16:28.058770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058754 5520 cpu_manager.go:414] "containerMap: removing container" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-web" Oct 31 03:16:28.058823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058772 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-web" Oct 31 03:16:28.058823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058786 5520 cpu_manager.go:414] "containerMap: removing container" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-metric" Oct 31 03:16:28.058823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058796 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-metric" Oct 31 03:16:28.058823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058808 5520 cpu_manager.go:414] "containerMap: removing container" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy" Oct 31 03:16:28.058823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058815 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058832 5520 cpu_manager.go:414] "containerMap: removing container" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="prom-label-proxy" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058841 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="prom-label-proxy" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058851 5520 cpu_manager.go:414] "containerMap: removing container" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="config-reloader" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058858 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="config-reloader" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058873 5520 cpu_manager.go:414] "containerMap: removing container" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="init-config-reloader" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058879 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="init-config-reloader" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058897 5520 cpu_manager.go:414] "containerMap: removing container" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="alertmanager" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058901 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="alertmanager" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058959 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-web" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058967 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy-metric" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058972 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="kube-rbac-proxy" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058977 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="prom-label-proxy" Oct 31 03:16:28.058979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058982 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="config-reloader" Oct 31 03:16:28.059403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058989 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="init-config-reloader" Oct 31 03:16:28.059403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.058995 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" containerName="alertmanager" Oct 31 03:16:28.068980 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.068958 5520 scope.go:122] "RemoveContainer" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.069282 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:28.069261 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": container with ID starting with 6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa not found: ID does not exist" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.069335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.069293 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} err="failed to get container status \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": rpc error: code = NotFound desc = could not find container \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": container with ID starting with 6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa not found: ID does not exist" Oct 31 03:16:28.069335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.069324 5520 scope.go:122] "RemoveContainer" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.069600 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:28.069583 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": container with ID starting with 0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe not found: ID does not exist" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.069644 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.069606 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} err="failed to get container status \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": rpc error: code = NotFound desc = could not find container \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": container with ID starting with 0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe not found: ID does not exist" Oct 31 03:16:28.069644 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.069622 5520 scope.go:122] "RemoveContainer" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.069865 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:28.069848 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": container with ID starting with 12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e not found: ID does not exist" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.069908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.069873 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} err="failed to get container status \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": rpc error: code = NotFound desc = could not find container \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": container with ID starting with 12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e not found: ID does not exist" Oct 31 03:16:28.069908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.069887 5520 scope.go:122] "RemoveContainer" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.070118 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:28.070104 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": container with ID starting with f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541 not found: ID does not exist" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.070161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070121 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} err="failed to get container status \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": rpc error: code = NotFound desc = could not find container \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": container with ID starting with f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541 not found: ID does not exist" Oct 31 03:16:28.070161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070132 5520 scope.go:122] "RemoveContainer" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.070332 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:28.070317 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": container with ID starting with 7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825 not found: ID does not exist" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.070374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070334 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} err="failed to get container status \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": rpc error: code = NotFound desc = could not find container \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": container with ID starting with 7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825 not found: ID does not exist" Oct 31 03:16:28.070374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070344 5520 scope.go:122] "RemoveContainer" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.070545 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:28.070529 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": container with ID starting with a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12 not found: ID does not exist" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.070585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070550 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} err="failed to get container status \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": rpc error: code = NotFound desc = could not find container \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": container with ID starting with a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12 not found: ID does not exist" Oct 31 03:16:28.070585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070563 5520 scope.go:122] "RemoveContainer" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.070760 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:28.070740 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": container with ID starting with 04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733 not found: ID does not exist" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.070848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070768 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733"} err="failed to get container status \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": rpc error: code = NotFound desc = could not find container \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": container with ID starting with 04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733 not found: ID does not exist" Oct 31 03:16:28.070848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.070787 5520 scope.go:122] "RemoveContainer" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.071043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071026 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} err="failed to get container status \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": rpc error: code = NotFound desc = could not find container \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": container with ID starting with 6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa not found: ID does not exist" Oct 31 03:16:28.071125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071045 5520 scope.go:122] "RemoveContainer" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.071310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071292 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} err="failed to get container status \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": rpc error: code = NotFound desc = could not find container \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": container with ID starting with 0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe not found: ID does not exist" Oct 31 03:16:28.071375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071311 5520 scope.go:122] "RemoveContainer" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.071507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071492 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} err="failed to get container status \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": rpc error: code = NotFound desc = could not find container \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": container with ID starting with 12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e not found: ID does not exist" Oct 31 03:16:28.071552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071507 5520 scope.go:122] "RemoveContainer" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.071730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071713 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} err="failed to get container status \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": rpc error: code = NotFound desc = could not find container \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": container with ID starting with f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541 not found: ID does not exist" Oct 31 03:16:28.071782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071731 5520 scope.go:122] "RemoveContainer" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.071944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071926 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} err="failed to get container status \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": rpc error: code = NotFound desc = could not find container \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": container with ID starting with 7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825 not found: ID does not exist" Oct 31 03:16:28.072013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.071947 5520 scope.go:122] "RemoveContainer" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.072176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072159 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} err="failed to get container status \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": rpc error: code = NotFound desc = could not find container \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": container with ID starting with a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12 not found: ID does not exist" Oct 31 03:16:28.072246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072177 5520 scope.go:122] "RemoveContainer" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.072382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072364 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733"} err="failed to get container status \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": rpc error: code = NotFound desc = could not find container \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": container with ID starting with 04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733 not found: ID does not exist" Oct 31 03:16:28.072444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072382 5520 scope.go:122] "RemoveContainer" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.072592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072574 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} err="failed to get container status \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": rpc error: code = NotFound desc = could not find container \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": container with ID starting with 6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa not found: ID does not exist" Oct 31 03:16:28.072636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072592 5520 scope.go:122] "RemoveContainer" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.072770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072751 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} err="failed to get container status \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": rpc error: code = NotFound desc = could not find container \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": container with ID starting with 0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe not found: ID does not exist" Oct 31 03:16:28.072836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072770 5520 scope.go:122] "RemoveContainer" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.072967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072946 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} err="failed to get container status \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": rpc error: code = NotFound desc = could not find container \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": container with ID starting with 12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e not found: ID does not exist" Oct 31 03:16:28.073012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.072969 5520 scope.go:122] "RemoveContainer" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.073162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073141 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} err="failed to get container status \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": rpc error: code = NotFound desc = could not find container \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": container with ID starting with f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541 not found: ID does not exist" Oct 31 03:16:28.073227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073162 5520 scope.go:122] "RemoveContainer" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.073409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073391 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} err="failed to get container status \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": rpc error: code = NotFound desc = could not find container \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": container with ID starting with 7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825 not found: ID does not exist" Oct 31 03:16:28.073480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073409 5520 scope.go:122] "RemoveContainer" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.073646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073631 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} err="failed to get container status \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": rpc error: code = NotFound desc = could not find container \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": container with ID starting with a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12 not found: ID does not exist" Oct 31 03:16:28.073692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073647 5520 scope.go:122] "RemoveContainer" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.073846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073829 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733"} err="failed to get container status \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": rpc error: code = NotFound desc = could not find container \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": container with ID starting with 04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733 not found: ID does not exist" Oct 31 03:16:28.073908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.073847 5520 scope.go:122] "RemoveContainer" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.074058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074041 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} err="failed to get container status \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": rpc error: code = NotFound desc = could not find container \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": container with ID starting with 6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa not found: ID does not exist" Oct 31 03:16:28.074110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074058 5520 scope.go:122] "RemoveContainer" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.074290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074273 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} err="failed to get container status \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": rpc error: code = NotFound desc = could not find container \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": container with ID starting with 0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe not found: ID does not exist" Oct 31 03:16:28.074360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074290 5520 scope.go:122] "RemoveContainer" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.074502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074484 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} err="failed to get container status \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": rpc error: code = NotFound desc = could not find container \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": container with ID starting with 12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e not found: ID does not exist" Oct 31 03:16:28.074502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074502 5520 scope.go:122] "RemoveContainer" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.074749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074729 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} err="failed to get container status \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": rpc error: code = NotFound desc = could not find container \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": container with ID starting with f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541 not found: ID does not exist" Oct 31 03:16:28.074795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074750 5520 scope.go:122] "RemoveContainer" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.074973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074953 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} err="failed to get container status \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": rpc error: code = NotFound desc = could not find container \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": container with ID starting with 7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825 not found: ID does not exist" Oct 31 03:16:28.075057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.074976 5520 scope.go:122] "RemoveContainer" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.075216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075186 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} err="failed to get container status \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": rpc error: code = NotFound desc = could not find container \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": container with ID starting with a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12 not found: ID does not exist" Oct 31 03:16:28.075262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075215 5520 scope.go:122] "RemoveContainer" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.075483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075465 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733"} err="failed to get container status \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": rpc error: code = NotFound desc = could not find container \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": container with ID starting with 04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733 not found: ID does not exist" Oct 31 03:16:28.075547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075484 5520 scope.go:122] "RemoveContainer" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.075724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075706 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} err="failed to get container status \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": rpc error: code = NotFound desc = could not find container \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": container with ID starting with 6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa not found: ID does not exist" Oct 31 03:16:28.075770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075725 5520 scope.go:122] "RemoveContainer" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.075933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075914 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} err="failed to get container status \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": rpc error: code = NotFound desc = could not find container \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": container with ID starting with 0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe not found: ID does not exist" Oct 31 03:16:28.076001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.075934 5520 scope.go:122] "RemoveContainer" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.076162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076132 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} err="failed to get container status \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": rpc error: code = NotFound desc = could not find container \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": container with ID starting with 12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e not found: ID does not exist" Oct 31 03:16:28.076221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076163 5520 scope.go:122] "RemoveContainer" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.076413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076396 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} err="failed to get container status \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": rpc error: code = NotFound desc = could not find container \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": container with ID starting with f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541 not found: ID does not exist" Oct 31 03:16:28.076476 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076414 5520 scope.go:122] "RemoveContainer" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.076629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076612 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} err="failed to get container status \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": rpc error: code = NotFound desc = could not find container \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": container with ID starting with 7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825 not found: ID does not exist" Oct 31 03:16:28.076677 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076630 5520 scope.go:122] "RemoveContainer" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.076827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076809 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} err="failed to get container status \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": rpc error: code = NotFound desc = could not find container \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": container with ID starting with a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12 not found: ID does not exist" Oct 31 03:16:28.076870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076826 5520 scope.go:122] "RemoveContainer" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.077010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.076995 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733"} err="failed to get container status \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": rpc error: code = NotFound desc = could not find container \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": container with ID starting with 04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733 not found: ID does not exist" Oct 31 03:16:28.077051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077012 5520 scope.go:122] "RemoveContainer" containerID="6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa" Oct 31 03:16:28.077219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077188 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa"} err="failed to get container status \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": rpc error: code = NotFound desc = could not find container \"6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa\": container with ID starting with 6da743b12f2e672408f436356cfce30cefffaf426cda53e765543e05a3c220fa not found: ID does not exist" Oct 31 03:16:28.077267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077220 5520 scope.go:122] "RemoveContainer" containerID="0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe" Oct 31 03:16:28.077424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077407 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe"} err="failed to get container status \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": rpc error: code = NotFound desc = could not find container \"0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe\": container with ID starting with 0d2067454b3f01d3818739b02c62766e3a751e848fc3ed05700390f3f5e0a5fe not found: ID does not exist" Oct 31 03:16:28.077494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077425 5520 scope.go:122] "RemoveContainer" containerID="12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e" Oct 31 03:16:28.077648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077633 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e"} err="failed to get container status \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": rpc error: code = NotFound desc = could not find container \"12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e\": container with ID starting with 12211350348ab7bf36d00073c222def2b4d551a943d41509a9672a951311a89e not found: ID does not exist" Oct 31 03:16:28.077686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077648 5520 scope.go:122] "RemoveContainer" containerID="f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541" Oct 31 03:16:28.077838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077822 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541"} err="failed to get container status \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": rpc error: code = NotFound desc = could not find container \"f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541\": container with ID starting with f8f024dafd5c8e098e7f75b6f73834973de39371e82ed0e5233ba261f6969541 not found: ID does not exist" Oct 31 03:16:28.077912 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.077839 5520 scope.go:122] "RemoveContainer" containerID="7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825" Oct 31 03:16:28.078042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.078023 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825"} err="failed to get container status \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": rpc error: code = NotFound desc = could not find container \"7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825\": container with ID starting with 7ac3d455e1266aee5df526c85753801a9246f813141e84a569d892a38c1c9825 not found: ID does not exist" Oct 31 03:16:28.078092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.078042 5520 scope.go:122] "RemoveContainer" containerID="a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12" Oct 31 03:16:28.078264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.078246 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12"} err="failed to get container status \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": rpc error: code = NotFound desc = could not find container \"a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12\": container with ID starting with a7668d791f9574b7024261fd7044958ef63a5354c1794597572b85aa59e3fa12 not found: ID does not exist" Oct 31 03:16:28.078338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.078265 5520 scope.go:122] "RemoveContainer" containerID="04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733" Oct 31 03:16:28.078461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.078443 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733"} err="failed to get container status \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": rpc error: code = NotFound desc = could not find container \"04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733\": container with ID starting with 04cb9a30a176bffed72bdb0b311179d937b968b3118086db21274426a7865733 not found: ID does not exist" Oct 31 03:16:28.086639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.086617 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:16:28.086745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.086730 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.091497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.091478 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 03:16:28.091597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.091492 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 03:16:28.091693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.091673 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 03:16:28.091746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.091679 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 03:16:28.092047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.092019 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 03:16:28.092123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.092066 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 03:16:28.092123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.092066 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 03:16:28.092562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.092526 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 03:16:28.092684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.092669 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7h7nd\"" Oct 31 03:16:28.092949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.092932 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 03:16:28.177254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177196 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177256 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177283 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c55d174-300d-40ef-ab2a-b8a6e5364b56-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177323 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c55d174-300d-40ef-ab2a-b8a6e5364b56-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177339 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c55d174-300d-40ef-ab2a-b8a6e5364b56-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177416 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-web-config\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177493 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c55d174-300d-40ef-ab2a-b8a6e5364b56-config-out\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177539 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177560 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177576 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177643 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177686 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c55d174-300d-40ef-ab2a-b8a6e5364b56-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.177766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.177716 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfzr\" (UniqueName: \"kubernetes.io/projected/7c55d174-300d-40ef-ab2a-b8a6e5364b56-kube-api-access-sqfzr\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278178 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278241 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278364 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c55d174-300d-40ef-ab2a-b8a6e5364b56-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278404 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c55d174-300d-40ef-ab2a-b8a6e5364b56-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278428 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c55d174-300d-40ef-ab2a-b8a6e5364b56-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278453 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-web-config\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278532 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c55d174-300d-40ef-ab2a-b8a6e5364b56-config-out\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278570 5520 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:16:28.278623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278618 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278650 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278678 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278720 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278743 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c55d174-300d-40ef-ab2a-b8a6e5364b56-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278770 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfzr\" (UniqueName: \"kubernetes.io/projected/7c55d174-300d-40ef-ab2a-b8a6e5364b56-kube-api-access-sqfzr\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.278837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.278770 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c55d174-300d-40ef-ab2a-b8a6e5364b56-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.279819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.279790 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c55d174-300d-40ef-ab2a-b8a6e5364b56-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.279981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.279956 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c55d174-300d-40ef-ab2a-b8a6e5364b56-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.281443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.281419 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c55d174-300d-40ef-ab2a-b8a6e5364b56-config-out\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.281582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.281562 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.281718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.281694 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-web-config\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.281988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.281966 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c55d174-300d-40ef-ab2a-b8a6e5364b56-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.282095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.281986 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.282507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.282482 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.282686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.282668 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.283397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.283380 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.283624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.283606 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c55d174-300d-40ef-ab2a-b8a6e5364b56-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.288476 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.288459 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfzr\" (UniqueName: \"kubernetes.io/projected/7c55d174-300d-40ef-ab2a-b8a6e5364b56-kube-api-access-sqfzr\") pod \"alertmanager-main-0\" (UID: \"7c55d174-300d-40ef-ab2a-b8a6e5364b56\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.487782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.487743 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 03:16:28.795674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:28.795637 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 03:16:28.799110 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:28.799078 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c55d174_300d_40ef_ab2a_b8a6e5364b56.slice/crio-d00b8fbb28a059d276ab34d402992ba394f80b6a2af36b0be27be26757c9f303 WatchSource:0}: Error finding container d00b8fbb28a059d276ab34d402992ba394f80b6a2af36b0be27be26757c9f303: Status 404 returned error can't find the container with id d00b8fbb28a059d276ab34d402992ba394f80b6a2af36b0be27be26757c9f303 Oct 31 03:16:29.010551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:29.010516 5520 generic.go:356] "Generic (PLEG): container finished" podID="7c55d174-300d-40ef-ab2a-b8a6e5364b56" containerID="c99c9cabb4ce9733e51df407a50dbf2c51f16aab78bac32061a6e3f5e0c09f90" exitCode=0 Oct 31 03:16:29.011096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:29.010552 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerDied","Data":"c99c9cabb4ce9733e51df407a50dbf2c51f16aab78bac32061a6e3f5e0c09f90"} Oct 31 03:16:29.011096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:29.010622 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerStarted","Data":"d00b8fbb28a059d276ab34d402992ba394f80b6a2af36b0be27be26757c9f303"} Oct 31 03:16:29.914068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:29.913994 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954e8b0e-c426-4cee-80d6-e296726215d4" path="/var/lib/kubelet/pods/954e8b0e-c426-4cee-80d6-e296726215d4/volumes" Oct 31 03:16:30.019620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:30.019586 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerStarted","Data":"8254c651129624838f7ad8ba86804a5c83942eb2e60ac2b6918dce10ca86688a"} Oct 31 03:16:30.019992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:30.019650 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerStarted","Data":"d832f3b089bff0204c0811e8ca73dcda56b8eba4eb2f40d8649ef94f100cb4df"} Oct 31 03:16:30.019992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:30.019677 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerStarted","Data":"1c409bfd02eede5605fb935ae16ba6183665b01e0705587c3876b96dd2664eb8"} Oct 31 03:16:30.019992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:30.019703 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerStarted","Data":"360384e552b07bf2832f77e504080915bc5f850eca83f85f9ceb2076650d17dd"} Oct 31 03:16:30.019992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:30.019725 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerStarted","Data":"41be0949a57d395a41510ef37b8ebb9ca25076c1cecad68d60ff261bbad71c87"} Oct 31 03:16:30.019992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:30.019749 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c55d174-300d-40ef-ab2a-b8a6e5364b56","Type":"ContainerStarted","Data":"b2ae618cdfc62265af5dbb5311498cc5374ad74d6edc7a1fdf41b30268e4c00d"} Oct 31 03:16:30.047076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:30.046984 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.034412921 podStartE2EDuration="2.046966423s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.553502ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:30.045677503 +0000 UTC m=+14.240200625" watchObservedRunningTime="2026-10-31 03:16:30.046966423 +0000 UTC m=+14.241489545" Oct 31 03:16:31.549401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.549370 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7548fffcd4-wq7fh"] Oct 31 03:16:31.552909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.552892 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.556879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.556846 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 03:16:31.556995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.556921 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 03:16:31.556995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.556963 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 03:16:31.557893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.557874 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 03:16:31.557893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.557893 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 03:16:31.558058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.557984 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q2ppx\"" Oct 31 03:16:31.560111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.560092 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 03:16:31.565629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.565608 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7548fffcd4-wq7fh"] Oct 31 03:16:31.607184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607154 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-secret-telemeter-client\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.607368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607232 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-serving-certs-ca-bundle\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.607368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607289 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.607464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607367 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-telemeter-client-tls\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.607464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607397 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-federate-client-tls\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.607464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607427 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-metrics-client-ca\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.607559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607517 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.607559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.607542 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5kpf\" (UniqueName: \"kubernetes.io/projected/3c24f704-898e-406b-8e06-d70e3fb4f65a-kube-api-access-m5kpf\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708159 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-telemeter-client-tls\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708194 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-federate-client-tls\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708253 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-metrics-client-ca\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708280 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708301 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5kpf\" (UniqueName: \"kubernetes.io/projected/3c24f704-898e-406b-8e06-d70e3fb4f65a-kube-api-access-m5kpf\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708351 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-secret-telemeter-client\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708386 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-serving-certs-ca-bundle\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.708449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.708406 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.709393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.709366 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-metrics-client-ca\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.709520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.709455 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-serving-certs-ca-bundle\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.709613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.709586 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c24f704-898e-406b-8e06-d70e3fb4f65a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.711114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.711090 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-telemeter-client-tls\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.711232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.711154 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-federate-client-tls\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.711352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.711335 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-secret-telemeter-client\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.711415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.711397 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c24f704-898e-406b-8e06-d70e3fb4f65a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.716111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.716093 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5kpf\" (UniqueName: \"kubernetes.io/projected/3c24f704-898e-406b-8e06-d70e3fb4f65a-kube-api-access-m5kpf\") pod \"telemeter-client-7548fffcd4-wq7fh\" (UID: \"3c24f704-898e-406b-8e06-d70e3fb4f65a\") " pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:31.869636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:31.869554 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" Oct 31 03:16:32.144355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:32.144283 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7548fffcd4-wq7fh"] Oct 31 03:16:32.147006 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:32.146975 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c24f704_898e_406b_8e06_d70e3fb4f65a.slice/crio-930e1747a745dd964f774c9f694c4783378a6de59b099ff566192e6ef598d01d WatchSource:0}: Error finding container 930e1747a745dd964f774c9f694c4783378a6de59b099ff566192e6ef598d01d: Status 404 returned error can't find the container with id 930e1747a745dd964f774c9f694c4783378a6de59b099ff566192e6ef598d01d Oct 31 03:16:32.149262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:32.149246 5520 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:16:33.031334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:33.031295 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" event={"ID":"3c24f704-898e-406b-8e06-d70e3fb4f65a","Type":"ContainerStarted","Data":"930e1747a745dd964f774c9f694c4783378a6de59b099ff566192e6ef598d01d"} Oct 31 03:16:35.040559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.040524 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" event={"ID":"3c24f704-898e-406b-8e06-d70e3fb4f65a","Type":"ContainerStarted","Data":"1b81a999911438b7e12b2112d8979af0302ba58c30ba27876e941e1a52747a6e"} Oct 31 03:16:35.040947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.040580 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" event={"ID":"3c24f704-898e-406b-8e06-d70e3fb4f65a","Type":"ContainerStarted","Data":"be3e4bbeb4929fceff20f76c2c838e0f9a8e712b2eee5465e3b83a016b7ad62b"} Oct 31 03:16:35.040947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.040599 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" event={"ID":"3c24f704-898e-406b-8e06-d70e3fb4f65a","Type":"ContainerStarted","Data":"27b4fa7efb2e006978527f6e74a4110fd36bfa613f425db6f36929c1fbe520b2"} Oct 31 03:16:35.062976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.062925 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7548fffcd4-wq7fh" podStartSLOduration=2.082356648 podStartE2EDuration="4.062912343s" totalImagesPullingTime="1.980555695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:35.062300062 +0000 UTC m=+19.256823194" watchObservedRunningTime="2026-10-31 03:16:35.062912343 +0000 UTC m=+19.257435463" Oct 31 03:16:35.145242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.145188 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d9c6c896-45rdm"] Oct 31 03:16:35.148589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.148569 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.164234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.164188 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d9c6c896-45rdm"] Oct 31 03:16:35.239112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239079 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-config\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.239112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239113 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-oauth-serving-cert\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.239369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239131 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-service-ca\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.239369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239233 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bn4b\" (UniqueName: \"kubernetes.io/projected/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-kube-api-access-6bn4b\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.239369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239341 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-tmp\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.239561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239391 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-serving-cert\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.239561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239419 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-oauth-config\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.239561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.239441 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-trusted-ca-bundle\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340065 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-tmp\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340106 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-serving-cert\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340124 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-oauth-config\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340156 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-trusted-ca-bundle\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340185 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-config\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340229 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-oauth-serving-cert\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340250 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-service-ca\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340383 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bn4b\" (UniqueName: \"kubernetes.io/projected/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-kube-api-access-6bn4b\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.340656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.340634 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-tmp\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.341032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.341005 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-service-ca\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.341138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.341005 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-config\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.341138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.341085 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-trusted-ca-bundle\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.341427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.341405 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-oauth-serving-cert\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.342828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.342801 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-serving-cert\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.342918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.342828 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-oauth-config\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.348863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.348841 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bn4b\" (UniqueName: \"kubernetes.io/projected/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-kube-api-access-6bn4b\") pod \"console-84d9c6c896-45rdm\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.459654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.459616 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:35.733349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.733315 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d9c6c896-45rdm"] Oct 31 03:16:35.737060 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:16:35.737034 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4def2d1e_f4bf_4084_8be3_02f2fea13f2f.slice/crio-4dd4d46a3da3cd818f111bdbead3dd1827a7170c5c20d038a9f78f83db12b948 WatchSource:0}: Error finding container 4dd4d46a3da3cd818f111bdbead3dd1827a7170c5c20d038a9f78f83db12b948: Status 404 returned error can't find the container with id 4dd4d46a3da3cd818f111bdbead3dd1827a7170c5c20d038a9f78f83db12b948 Oct 31 03:16:35.961253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:35.961228 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t5fpc" Oct 31 03:16:36.044721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:36.044632 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9c6c896-45rdm" event={"ID":"4def2d1e-f4bf-4084-8be3-02f2fea13f2f","Type":"ContainerStarted","Data":"d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e"} Oct 31 03:16:36.044721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:36.044713 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9c6c896-45rdm" event={"ID":"4def2d1e-f4bf-4084-8be3-02f2fea13f2f","Type":"ContainerStarted","Data":"4dd4d46a3da3cd818f111bdbead3dd1827a7170c5c20d038a9f78f83db12b948"} Oct 31 03:16:36.062977 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:36.062920 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-84d9c6c896-45rdm" podStartSLOduration=1.06290867 podStartE2EDuration="1.06290867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:36.062001406 +0000 UTC m=+20.256524537" watchObservedRunningTime="2026-10-31 03:16:36.06290867 +0000 UTC m=+20.257431791" Oct 31 03:16:39.356873 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:16:39.356837 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:16:41.907913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:41.907844 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66fc4df878-chnw2" podUID="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" containerName="console" containerID="cri-o://ab86b2318db991b58936225bbb5646ea6e8deb441833d5ac1655cfc7881b6f7c" gracePeriod=15 Oct 31 03:16:42.068260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.068222 5520 generic.go:356] "Generic (PLEG): container finished" podID="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" containerID="ab86b2318db991b58936225bbb5646ea6e8deb441833d5ac1655cfc7881b6f7c" exitCode=0 Oct 31 03:16:42.068260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.068235 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc4df878-chnw2" event={"ID":"0fbe170d-bfcb-4e38-8182-59fdcb8b1628","Type":"ContainerDied","Data":"ab86b2318db991b58936225bbb5646ea6e8deb441833d5ac1655cfc7881b6f7c"} Oct 31 03:16:42.151784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.151761 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:42.235836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.235809 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8z9f4" Oct 31 03:16:42.299996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.299965 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\") pod \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " Oct 31 03:16:42.299996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300000 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\") pod \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " Oct 31 03:16:42.300250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300039 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\") pod \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " Oct 31 03:16:42.300250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300067 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\") pod \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " Oct 31 03:16:42.300250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300092 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\") pod \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " Oct 31 03:16:42.300250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300136 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\") pod \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " Oct 31 03:16:42.300250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300180 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\" (UniqueName: \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\") pod \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\" (UID: \"0fbe170d-bfcb-4e38-8182-59fdcb8b1628\") " Oct 31 03:16:42.300507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300476 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp" pod "0fbe170d-bfcb-4e38-8182-59fdcb8b1628" (UID: "0fbe170d-bfcb-4e38-8182-59fdcb8b1628"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:16:42.300595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300526 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca" pod "0fbe170d-bfcb-4e38-8182-59fdcb8b1628" (UID: "0fbe170d-bfcb-4e38-8182-59fdcb8b1628"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:42.300660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300592 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config" pod "0fbe170d-bfcb-4e38-8182-59fdcb8b1628" (UID: "0fbe170d-bfcb-4e38-8182-59fdcb8b1628"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:42.300717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.300694 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert" pod "0fbe170d-bfcb-4e38-8182-59fdcb8b1628" (UID: "0fbe170d-bfcb-4e38-8182-59fdcb8b1628"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:42.302567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.302536 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config" pod "0fbe170d-bfcb-4e38-8182-59fdcb8b1628" (UID: "0fbe170d-bfcb-4e38-8182-59fdcb8b1628"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:42.302669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.302608 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm" pod "0fbe170d-bfcb-4e38-8182-59fdcb8b1628" (UID: "0fbe170d-bfcb-4e38-8182-59fdcb8b1628"). InnerVolumeSpecName "kube-api-access-j9vnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:42.302669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.302621 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert" pod "0fbe170d-bfcb-4e38-8182-59fdcb8b1628" (UID: "0fbe170d-bfcb-4e38-8182-59fdcb8b1628"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:42.401813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.401758 5520 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-tmp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.401813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.401809 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vnm\" (UniqueName: \"kubernetes.io/projected/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-kube-api-access-j9vnm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.401813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.401825 5520 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.402104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.401839 5520 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.402104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.401856 5520 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-oauth-serving-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.402104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.401869 5520 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-serving-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.402104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.401885 5520 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0fbe170d-bfcb-4e38-8182-59fdcb8b1628-console-oauth-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:42.641283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:42.641187 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbv64_4080580a-a99a-49e0-a255-06cb08581290/serve-healthcheck-canary/0.log" Oct 31 03:16:43.073535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:43.073499 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66fc4df878-chnw2" Oct 31 03:16:43.073943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:43.073498 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66fc4df878-chnw2" event={"ID":"0fbe170d-bfcb-4e38-8182-59fdcb8b1628","Type":"ContainerDied","Data":"fd3bc0faaada5f46e43e5e33b15b1e5f1ec7e8772b074e5609e45481a2223fed"} Oct 31 03:16:43.073943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:43.073623 5520 scope.go:122] "RemoveContainer" containerID="ab86b2318db991b58936225bbb5646ea6e8deb441833d5ac1655cfc7881b6f7c" Oct 31 03:16:43.094632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:43.094604 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66fc4df878-chnw2"] Oct 31 03:16:43.098339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:43.098319 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66fc4df878-chnw2"] Oct 31 03:16:43.914250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:43.914195 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" path="/var/lib/kubelet/pods/0fbe170d-bfcb-4e38-8182-59fdcb8b1628/volumes" Oct 31 03:16:45.459869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:45.459827 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:45.460293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:45.459914 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:45.464723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:45.464697 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:46.091847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:46.091822 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:16:46.137956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:46.137926 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4c575df7-h42hr"] Oct 31 03:16:48.219926 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:16:48.219896 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dlvjw" Oct 31 03:17:09.357001 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:17:09.356954 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:17:11.158447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.158398 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f4c575df7-h42hr" podUID="a44a09c8-5e56-478e-b27f-eca60624a37b" containerName="console" containerID="cri-o://1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4" gracePeriod=15 Oct 31 03:17:11.178764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.178736 5520 patch_prober.go:29] interesting pod/console-7f4c575df7-h42hr container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" start-of-body= Oct 31 03:17:11.178868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.178802 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-7f4c575df7-h42hr" podUID="a44a09c8-5e56-478e-b27f-eca60624a37b" containerName="console" probeResult="failure" output="Get \"https://10.134.0.14:8443/health\": dial tcp 10.134.0.14:8443: connect: connection refused" Oct 31 03:17:11.402957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.402933 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:17:11.423951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.423873 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.423951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.423915 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.423951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.423954 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.424245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.423979 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.424245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.423997 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\" (UniqueName: \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.424245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424025 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.424245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424122 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.424245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424170 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\") pod \"a44a09c8-5e56-478e-b27f-eca60624a37b\" (UID: \"a44a09c8-5e56-478e-b27f-eca60624a37b\") " Oct 31 03:17:11.424539 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424510 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:11.424539 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424523 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:11.424760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424734 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:11.424760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424740 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:17:11.424916 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.424656 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:17:11.426643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.426586 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:11.428031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.427952 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "kube-api-access-5pxcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:17:11.428288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.428258 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert" pod "a44a09c8-5e56-478e-b27f-eca60624a37b" (UID: "a44a09c8-5e56-478e-b27f-eca60624a37b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:17:11.525259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525226 5520 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-oauth-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:11.525259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525253 5520 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a44a09c8-5e56-478e-b27f-eca60624a37b-console-serving-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:11.525259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525265 5520 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-console-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:11.525491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525274 5520 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-trusted-ca-bundle\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:11.525491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525283 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pxcr\" (UniqueName: \"kubernetes.io/projected/a44a09c8-5e56-478e-b27f-eca60624a37b-kube-api-access-5pxcr\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:11.525491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525294 5520 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a44a09c8-5e56-478e-b27f-eca60624a37b-tmp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:11.525491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525303 5520 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-oauth-serving-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:11.525491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:11.525311 5520 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a44a09c8-5e56-478e-b27f-eca60624a37b-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:17:12.187222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.187172 5520 generic.go:356] "Generic (PLEG): container finished" podID="a44a09c8-5e56-478e-b27f-eca60624a37b" containerID="1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4" exitCode=0 Oct 31 03:17:12.187626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.187262 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4c575df7-h42hr" event={"ID":"a44a09c8-5e56-478e-b27f-eca60624a37b","Type":"ContainerDied","Data":"1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4"} Oct 31 03:17:12.187626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.187297 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4c575df7-h42hr" Oct 31 03:17:12.187626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.187330 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4c575df7-h42hr" event={"ID":"a44a09c8-5e56-478e-b27f-eca60624a37b","Type":"ContainerDied","Data":"c6f7b9afe84d1aad0db4f98f0d3899e0bc7032f3084a89fd8699c6de2b87f554"} Oct 31 03:17:12.187626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.187357 5520 scope.go:122] "RemoveContainer" containerID="1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4" Oct 31 03:17:12.195571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.195555 5520 scope.go:122] "RemoveContainer" containerID="1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4" Oct 31 03:17:12.195859 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:17:12.195830 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4\": container with ID starting with 1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4 not found: ID does not exist" containerID="1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4" Oct 31 03:17:12.195943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.195856 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4"} err="failed to get container status \"1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4\": rpc error: code = NotFound desc = could not find container \"1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4\": container with ID starting with 1d5193e44b9fb8d5701ff0927315ec63510e6d5cf871565465cc3921415177e4 not found: ID does not exist" Oct 31 03:17:12.205357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.205337 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4c575df7-h42hr"] Oct 31 03:17:12.209554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:12.209536 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f4c575df7-h42hr"] Oct 31 03:17:13.914351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:13.914319 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44a09c8-5e56-478e-b27f-eca60624a37b" path="/var/lib/kubelet/pods/a44a09c8-5e56-478e-b27f-eca60624a37b/volumes" Oct 31 03:17:38.267736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.267698 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6974f696-kqrj2"] Oct 31 03:17:38.268175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.268092 5520 cpu_manager.go:414] "containerMap: removing container" podUID="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" containerName="console" Oct 31 03:17:38.268175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.268112 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" containerName="console" Oct 31 03:17:38.268175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.268137 5520 cpu_manager.go:414] "containerMap: removing container" podUID="a44a09c8-5e56-478e-b27f-eca60624a37b" containerName="console" Oct 31 03:17:38.268175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.268146 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44a09c8-5e56-478e-b27f-eca60624a37b" containerName="console" Oct 31 03:17:38.268395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.268233 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbe170d-bfcb-4e38-8182-59fdcb8b1628" containerName="console" Oct 31 03:17:38.268395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.268251 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44a09c8-5e56-478e-b27f-eca60624a37b" containerName="console" Oct 31 03:17:38.271443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.271415 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.283757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.283731 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6974f696-kqrj2"] Oct 31 03:17:38.333495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333461 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-oauth-serving-cert\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.333759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333513 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-service-ca\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.333759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333531 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-trusted-ca-bundle\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.333759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333606 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-config\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.333759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333646 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-serving-cert\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.333759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333705 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-oauth-config\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.333957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333802 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-tmp\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.333957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.333838 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x6p8\" (UniqueName: \"kubernetes.io/projected/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-kube-api-access-2x6p8\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434461 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-oauth-serving-cert\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434514 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-service-ca\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434579 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-trusted-ca-bundle\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434617 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-config\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434638 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-serving-cert\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434652 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-oauth-config\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434688 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-tmp\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.434939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.434720 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x6p8\" (UniqueName: \"kubernetes.io/projected/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-kube-api-access-2x6p8\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.435311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.435290 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-service-ca\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.435410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.435328 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-tmp\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.435578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.435559 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-oauth-serving-cert\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.435735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.435713 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-config\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.436028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.436003 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-trusted-ca-bundle\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.437330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.437307 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-oauth-config\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.437422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.437354 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-console-serving-cert\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.443180 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.443155 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x6p8\" (UniqueName: \"kubernetes.io/projected/79b1ce9a-51ac-42a2-b4f7-08ba651898c8-kube-api-access-2x6p8\") pod \"console-5f6974f696-kqrj2\" (UID: \"79b1ce9a-51ac-42a2-b4f7-08ba651898c8\") " pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.581799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.581710 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:38.854330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:38.854254 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6974f696-kqrj2"] Oct 31 03:17:38.857894 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:17:38.857866 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b1ce9a_51ac_42a2_b4f7_08ba651898c8.slice/crio-1f200ab9a508fa6dbca487ae20966558c83138ebe09c1922ec609be705a5bac5 WatchSource:0}: Error finding container 1f200ab9a508fa6dbca487ae20966558c83138ebe09c1922ec609be705a5bac5: Status 404 returned error can't find the container with id 1f200ab9a508fa6dbca487ae20966558c83138ebe09c1922ec609be705a5bac5 Oct 31 03:17:39.276288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:39.276257 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6974f696-kqrj2" event={"ID":"79b1ce9a-51ac-42a2-b4f7-08ba651898c8","Type":"ContainerStarted","Data":"516c3e2eb852dab2401a999e59e0cdde93613853b6848a4c489bebffd38f2851"} Oct 31 03:17:39.276665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:39.276303 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6974f696-kqrj2" event={"ID":"79b1ce9a-51ac-42a2-b4f7-08ba651898c8","Type":"ContainerStarted","Data":"1f200ab9a508fa6dbca487ae20966558c83138ebe09c1922ec609be705a5bac5"} Oct 31 03:17:39.295728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:39.295676 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f6974f696-kqrj2" podStartSLOduration=1.295661535 podStartE2EDuration="1.295661535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:39.294155603 +0000 UTC m=+83.488678768" watchObservedRunningTime="2026-10-31 03:17:39.295661535 +0000 UTC m=+83.490184655" Oct 31 03:17:39.357603 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:17:39.357563 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:17:48.582790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:48.582755 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:48.583412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:48.582825 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:48.587581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:48.587559 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:49.315236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:49.315186 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f6974f696-kqrj2" Oct 31 03:17:49.360896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:17:49.360863 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d9c6c896-45rdm"] Oct 31 03:18:09.356472 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:18:09.356387 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:18:14.381344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.381285 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-84d9c6c896-45rdm" podUID="4def2d1e-f4bf-4084-8be3-02f2fea13f2f" containerName="console" containerID="cri-o://d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e" gracePeriod=15 Oct 31 03:18:14.624021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.623999 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:18:14.733291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733257 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-config\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-config\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733297 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-tmp\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733327 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-trusted-ca-bundle\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733364 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-oauth-serving-cert\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733437 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-service-ca\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-service-ca\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733462 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-serving-cert\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733500 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-oauth-config\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733531 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-kube-api-access-6bn4b\" (UniqueName: \"kubernetes.io/projected/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-kube-api-access-6bn4b\") pod \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\" (UID: \"4def2d1e-f4bf-4084-8be3-02f2fea13f2f\") " Oct 31 03:18:14.733738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733603 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-tmp" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:18:14.733891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733819 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-config" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:14.733891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733837 5520 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-tmp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.733995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733887 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-oauth-serving-cert" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:14.733995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.733950 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-trusted-ca-bundle" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:14.734124 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.734098 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-service-ca" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:18:14.735885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.735865 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-kube-api-access-6bn4b" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "kube-api-access-6bn4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:18:14.735968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.735885 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-oauth-config" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:14.735968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.735928 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-serving-cert" pod "4def2d1e-f4bf-4084-8be3-02f2fea13f2f" (UID: "4def2d1e-f4bf-4084-8be3-02f2fea13f2f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:18:14.835271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.835233 5520 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.835271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.835264 5520 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-serving-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.835271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.835273 5520 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-oauth-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.835271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.835282 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bn4b\" (UniqueName: \"kubernetes.io/projected/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-kube-api-access-6bn4b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.835514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.835292 5520 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-console-config\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.835514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.835300 5520 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-trusted-ca-bundle\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:14.835514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:14.835309 5520 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4def2d1e-f4bf-4084-8be3-02f2fea13f2f-oauth-serving-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:18:15.408036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.407999 5520 generic.go:356] "Generic (PLEG): container finished" podID="4def2d1e-f4bf-4084-8be3-02f2fea13f2f" containerID="d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e" exitCode=0 Oct 31 03:18:15.408523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.408050 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d9c6c896-45rdm" Oct 31 03:18:15.408523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.408083 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9c6c896-45rdm" event={"ID":"4def2d1e-f4bf-4084-8be3-02f2fea13f2f","Type":"ContainerDied","Data":"d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e"} Oct 31 03:18:15.408523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.408145 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d9c6c896-45rdm" event={"ID":"4def2d1e-f4bf-4084-8be3-02f2fea13f2f","Type":"ContainerDied","Data":"4dd4d46a3da3cd818f111bdbead3dd1827a7170c5c20d038a9f78f83db12b948"} Oct 31 03:18:15.408523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.408173 5520 scope.go:122] "RemoveContainer" containerID="d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e" Oct 31 03:18:15.416572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.416557 5520 scope.go:122] "RemoveContainer" containerID="d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e" Oct 31 03:18:15.416835 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:18:15.416818 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e\": container with ID starting with d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e not found: ID does not exist" containerID="d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e" Oct 31 03:18:15.416884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.416843 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e"} err="failed to get container status \"d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e\": rpc error: code = NotFound desc = could not find container \"d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e\": container with ID starting with d3f3de6645523ef4f340567c05ad1c911df1cd0c1ab4add20e5f0c4390caf04e not found: ID does not exist" Oct 31 03:18:15.428520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.428501 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d9c6c896-45rdm"] Oct 31 03:18:15.430592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.430572 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d9c6c896-45rdm"] Oct 31 03:18:15.918894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:18:15.918867 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4def2d1e-f4bf-4084-8be3-02f2fea13f2f" path="/var/lib/kubelet/pods/4def2d1e-f4bf-4084-8be3-02f2fea13f2f/volumes" Oct 31 03:18:39.356454 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:18:39.356419 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:19:09.357261 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:19:09.357219 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:19:39.356649 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:19:39.356568 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:20:04.849237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.849190 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz"] Oct 31 03:20:04.851712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.849483 5520 cpu_manager.go:414] "containerMap: removing container" podUID="4def2d1e-f4bf-4084-8be3-02f2fea13f2f" containerName="console" Oct 31 03:20:04.851712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.849493 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4def2d1e-f4bf-4084-8be3-02f2fea13f2f" containerName="console" Oct 31 03:20:04.851712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.849567 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="4def2d1e-f4bf-4084-8be3-02f2fea13f2f" containerName="console" Oct 31 03:20:04.852508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.852492 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.856612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.856589 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 03:20:04.856756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.856740 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 03:20:04.869471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.869453 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz"] Oct 31 03:20:04.884184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.884162 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrct\" (UniqueName: \"kubernetes.io/projected/3e63291d-c94f-490e-80c1-8df85450c40f-kube-api-access-fdrct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.884329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.884234 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.884329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.884310 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.942684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.942660 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7s88r\"" Oct 31 03:20:04.985041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.985010 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrct\" (UniqueName: \"kubernetes.io/projected/3e63291d-c94f-490e-80c1-8df85450c40f-kube-api-access-fdrct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.985161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.985066 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.985161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.985099 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.985491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.985471 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.985557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.985503 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:04.994188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:04.994170 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrct\" (UniqueName: \"kubernetes.io/projected/3e63291d-c94f-490e-80c1-8df85450c40f-kube-api-access-fdrct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:05.155525 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.155439 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b"] Oct 31 03:20:05.158641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.158619 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.162716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.162695 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:20:05.163348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.163332 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:20:05.163426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.163356 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:20:05.163426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.163384 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hwjnd\"" Oct 31 03:20:05.167432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.167404 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:05.167529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.167483 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:20:05.167591 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.167522 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:20:05.167667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.167653 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:20:05.175032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.175005 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b"] Oct 31 03:20:05.186792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.186753 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.186907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.186858 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.186907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.186899 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.187029 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.186933 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.187029 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.186992 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.187029 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.187016 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6b2c\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-kube-api-access-m6b2c\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.187225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.187060 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.288298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.288178 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.288298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.288225 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6b2c\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-kube-api-access-m6b2c\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.288298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.288261 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.288298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.288301 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.288654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.288327 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.288654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.288352 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.288654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.288381 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.289188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.289159 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.290638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.290606 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.290768 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.290749 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.290824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.290766 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.290824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.290791 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.295955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.295938 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6b2c\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-kube-api-access-m6b2c\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.298179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.298157 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-62d4b\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.443769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.443740 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz"] Oct 31 03:20:05.447496 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:20:05.447472 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e63291d_c94f_490e_80c1_8df85450c40f.slice/crio-9131bb3c665e3df9bafefe9aff013530efd75150df9474f8c2d962ee06830063 WatchSource:0}: Error finding container 9131bb3c665e3df9bafefe9aff013530efd75150df9474f8c2d962ee06830063: Status 404 returned error can't find the container with id 9131bb3c665e3df9bafefe9aff013530efd75150df9474f8c2d962ee06830063 Oct 31 03:20:05.470645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.470624 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:05.749238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.749149 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b"] Oct 31 03:20:05.754619 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:20:05.754594 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c8bb12_78d0_4cb7_8841_26b91d1a3099.slice/crio-b4b88b601a793242e548d692dede826e04f7f72d413a08ccfd58b3647d267473 WatchSource:0}: Error finding container b4b88b601a793242e548d692dede826e04f7f72d413a08ccfd58b3647d267473: Status 404 returned error can't find the container with id b4b88b601a793242e548d692dede826e04f7f72d413a08ccfd58b3647d267473 Oct 31 03:20:05.780224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.780174 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" event={"ID":"3e63291d-c94f-490e-80c1-8df85450c40f","Type":"ContainerStarted","Data":"9131bb3c665e3df9bafefe9aff013530efd75150df9474f8c2d962ee06830063"} Oct 31 03:20:05.781281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:05.781253 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" event={"ID":"e9c8bb12-78d0-4cb7-8841-26b91d1a3099","Type":"ContainerStarted","Data":"b4b88b601a793242e548d692dede826e04f7f72d413a08ccfd58b3647d267473"} Oct 31 03:20:09.360348 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:20:09.360311 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:20:09.502195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:09.502166 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:09.502279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:09.502244 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:20:09.798766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:09.798726 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" event={"ID":"e9c8bb12-78d0-4cb7-8841-26b91d1a3099","Type":"ContainerStarted","Data":"6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414"} Oct 31 03:20:09.798953 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:09.798868 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:09.832834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:09.832100 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" podStartSLOduration=1.087311593 podStartE2EDuration="4.832083538s" totalImagesPullingTime="3.744771945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:09.831417054 +0000 UTC m=+234.025940193" watchObservedRunningTime="2026-10-31 03:20:09.832083538 +0000 UTC m=+234.026606660" Oct 31 03:20:10.804640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:10.804618 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:20:14.822901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:14.822863 5520 generic.go:356] "Generic (PLEG): container finished" podID="3e63291d-c94f-490e-80c1-8df85450c40f" containerID="7f710e3c8d65eefc76712bb65d1536a3f7147ed988ef29af2cce6ac04d06d027" exitCode=0 Oct 31 03:20:14.823326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:14.822951 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" event={"ID":"3e63291d-c94f-490e-80c1-8df85450c40f","Type":"ContainerDied","Data":"7f710e3c8d65eefc76712bb65d1536a3f7147ed988ef29af2cce6ac04d06d027"} Oct 31 03:20:15.830687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:15.830604 5520 generic.go:356] "Generic (PLEG): container finished" podID="3e63291d-c94f-490e-80c1-8df85450c40f" containerID="9190ece83fb136fd2409450d4a60a3caafa199df9483df164661d9be43f1e062" exitCode=0 Oct 31 03:20:15.830687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:15.830664 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" event={"ID":"3e63291d-c94f-490e-80c1-8df85450c40f","Type":"ContainerDied","Data":"9190ece83fb136fd2409450d4a60a3caafa199df9483df164661d9be43f1e062"} Oct 31 03:20:22.860979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:22.860943 5520 generic.go:356] "Generic (PLEG): container finished" podID="3e63291d-c94f-490e-80c1-8df85450c40f" containerID="94741cad81d1259f89ca7a53344d7bcfc70d7111c1d5a31dddcd9f0600c8e413" exitCode=0 Oct 31 03:20:22.861414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:22.860994 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" event={"ID":"3e63291d-c94f-490e-80c1-8df85450c40f","Type":"ContainerDied","Data":"94741cad81d1259f89ca7a53344d7bcfc70d7111c1d5a31dddcd9f0600c8e413"} Oct 31 03:20:23.987920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:23.987898 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:24.051347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.051316 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e63291d-c94f-490e-80c1-8df85450c40f-kube-api-access-fdrct\" (UniqueName: \"kubernetes.io/projected/3e63291d-c94f-490e-80c1-8df85450c40f-kube-api-access-fdrct\") pod \"3e63291d-c94f-490e-80c1-8df85450c40f\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " Oct 31 03:20:24.051347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.051350 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-bundle\") pod \"3e63291d-c94f-490e-80c1-8df85450c40f\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " Oct 31 03:20:24.051562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.051424 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-util\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-util\") pod \"3e63291d-c94f-490e-80c1-8df85450c40f\" (UID: \"3e63291d-c94f-490e-80c1-8df85450c40f\") " Oct 31 03:20:24.051958 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.051897 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-bundle" pod "3e63291d-c94f-490e-80c1-8df85450c40f" (UID: "3e63291d-c94f-490e-80c1-8df85450c40f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:24.053595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.053568 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e63291d-c94f-490e-80c1-8df85450c40f-kube-api-access-fdrct" pod "3e63291d-c94f-490e-80c1-8df85450c40f" (UID: "3e63291d-c94f-490e-80c1-8df85450c40f"). InnerVolumeSpecName "kube-api-access-fdrct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:24.055940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.055918 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-util" pod "3e63291d-c94f-490e-80c1-8df85450c40f" (UID: "3e63291d-c94f-490e-80c1-8df85450c40f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:24.152176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.152100 5520 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-util\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:24.152176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.152128 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdrct\" (UniqueName: \"kubernetes.io/projected/3e63291d-c94f-490e-80c1-8df85450c40f-kube-api-access-fdrct\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:24.152176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.152137 5520 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e63291d-c94f-490e-80c1-8df85450c40f-bundle\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:24.870156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.870121 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" event={"ID":"3e63291d-c94f-490e-80c1-8df85450c40f","Type":"ContainerDied","Data":"9131bb3c665e3df9bafefe9aff013530efd75150df9474f8c2d962ee06830063"} Oct 31 03:20:24.870348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.870167 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9131bb3c665e3df9bafefe9aff013530efd75150df9474f8c2d962ee06830063" Oct 31 03:20:24.870348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:24.870133 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rcbrz" Oct 31 03:20:39.356894 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:20:39.356855 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:20:43.349292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349256 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2"] Oct 31 03:20:43.349695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349555 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3e63291d-c94f-490e-80c1-8df85450c40f" containerName="extract" Oct 31 03:20:43.349695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349566 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e63291d-c94f-490e-80c1-8df85450c40f" containerName="extract" Oct 31 03:20:43.349695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349583 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3e63291d-c94f-490e-80c1-8df85450c40f" containerName="util" Oct 31 03:20:43.349695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349588 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e63291d-c94f-490e-80c1-8df85450c40f" containerName="util" Oct 31 03:20:43.349695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349600 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3e63291d-c94f-490e-80c1-8df85450c40f" containerName="pull" Oct 31 03:20:43.349695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349605 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e63291d-c94f-490e-80c1-8df85450c40f" containerName="pull" Oct 31 03:20:43.349695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.349660 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e63291d-c94f-490e-80c1-8df85450c40f" containerName="extract" Oct 31 03:20:43.353669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.353653 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.378621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.378589 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2"] Oct 31 03:20:43.429333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.429303 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:20:43.429682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.429630 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:20:43.429834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.429820 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:20:43.429895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.429825 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:20:43.429895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.429843 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dw2ct\"" Oct 31 03:20:43.429895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.429836 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:20:43.518938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.518905 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7vt\" (UniqueName: \"kubernetes.io/projected/f7fdbf06-a4c6-4718-b36d-6e42eceab645-kube-api-access-xh7vt\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.519146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.518953 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fdbf06-a4c6-4718-b36d-6e42eceab645-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.519146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.518975 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.519146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.519098 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7fdbf06-a4c6-4718-b36d-6e42eceab645-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.519146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.519133 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.519318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.519197 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.619780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.619711 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.619780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.619754 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7vt\" (UniqueName: \"kubernetes.io/projected/f7fdbf06-a4c6-4718-b36d-6e42eceab645-kube-api-access-xh7vt\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.619944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.619795 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fdbf06-a4c6-4718-b36d-6e42eceab645-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.619944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.619826 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.619944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.619865 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7fdbf06-a4c6-4718-b36d-6e42eceab645-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.619944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.619899 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.620385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.620357 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7fdbf06-a4c6-4718-b36d-6e42eceab645-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.622433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.622404 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.622537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.622465 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.622682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.622663 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7fdbf06-a4c6-4718-b36d-6e42eceab645-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.622861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.622846 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7fdbf06-a4c6-4718-b36d-6e42eceab645-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.628596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.628577 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7vt\" (UniqueName: \"kubernetes.io/projected/f7fdbf06-a4c6-4718-b36d-6e42eceab645-kube-api-access-xh7vt\") pod \"opendatahub-operator-controller-manager-55975df5cc-rx6m2\" (UID: \"f7fdbf06-a4c6-4718-b36d-6e42eceab645\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.666098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.666070 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:43.958610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:43.958580 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2"] Oct 31 03:20:43.962172 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:20:43.962145 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fdbf06_a4c6_4718_b36d_6e42eceab645.slice/crio-08c9e5826590311f3d55dbd8b7faa5ec5b0743bb6e7ea94f94a95b1e70a3c14a WatchSource:0}: Error finding container 08c9e5826590311f3d55dbd8b7faa5ec5b0743bb6e7ea94f94a95b1e70a3c14a: Status 404 returned error can't find the container with id 08c9e5826590311f3d55dbd8b7faa5ec5b0743bb6e7ea94f94a95b1e70a3c14a Oct 31 03:20:44.945378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:44.945341 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" event={"ID":"f7fdbf06-a4c6-4718-b36d-6e42eceab645","Type":"ContainerStarted","Data":"08c9e5826590311f3d55dbd8b7faa5ec5b0743bb6e7ea94f94a95b1e70a3c14a"} Oct 31 03:20:47.959983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:47.959948 5520 generic.go:356] "Generic (PLEG): container finished" podID="f7fdbf06-a4c6-4718-b36d-6e42eceab645" containerID="9426cd3a1003f962633c16d6a9c12a7e1d859059a822adfda718f8c25784ba1e" exitCode=0 Oct 31 03:20:47.960480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:47.960031 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" event={"ID":"f7fdbf06-a4c6-4718-b36d-6e42eceab645","Type":"ContainerDied","Data":"9426cd3a1003f962633c16d6a9c12a7e1d859059a822adfda718f8c25784ba1e"} Oct 31 03:20:48.965795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:48.965757 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" event={"ID":"f7fdbf06-a4c6-4718-b36d-6e42eceab645","Type":"ContainerStarted","Data":"0a7b7c188e26bdd8ffbebe693b41a8ebf0232b4127413cc904353e287743aaf6"} Oct 31 03:20:48.966260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:48.965895 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:20:49.022024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:49.021966 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" podStartSLOduration=2.114855583 podStartE2EDuration="6.021949736s" totalImagesPullingTime="3.422477509s" totalInitContainerRuntime="484.616644ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:49.020017813 +0000 UTC m=+273.214540936" watchObservedRunningTime="2026-10-31 03:20:49.021949736 +0000 UTC m=+273.216472856" Oct 31 03:20:59.971741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:20:59.971660 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-rx6m2" Oct 31 03:21:09.357512 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:21:09.357473 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:21:12.235937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.235903 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5"] Oct 31 03:21:12.239411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.239388 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.245758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.245734 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 03:21:12.245897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.245742 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:21:12.245897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.245799 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pc69c\"" Oct 31 03:21:12.252512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.252490 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5"] Oct 31 03:21:12.366610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.366579 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.366771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.366623 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tmp\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.366771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.366726 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvsts\" (UniqueName: \"kubernetes.io/projected/8aec38a2-ea3c-4bc3-a971-10953e2299e5-kube-api-access-fvsts\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.467436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.467398 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.467628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.467458 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tmp\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.467628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.467512 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvsts\" (UniqueName: \"kubernetes.io/projected/8aec38a2-ea3c-4bc3-a971-10953e2299e5-kube-api-access-fvsts\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.467628 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:21:12.467556 5520 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 31 03:21:12.467809 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:21:12.467644 5520 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tls-certs podName:8aec38a2-ea3c-4bc3-a971-10953e2299e5 nodeName:}" failed. No retries permitted until 2026-10-31 03:21:12.967625948 +0000 UTC m=+297.162149066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tls-certs") pod "kube-auth-proxy-86bb8bf66f-zt5k5" (UID: "8aec38a2-ea3c-4bc3-a971-10953e2299e5") : secret "kube-auth-proxy-tls" not found Oct 31 03:21:12.469766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.469746 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tmp\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.514013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.513943 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvsts\" (UniqueName: \"kubernetes.io/projected/8aec38a2-ea3c-4bc3-a971-10953e2299e5-kube-api-access-fvsts\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.979390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.979356 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:12.982119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:12.982089 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aec38a2-ea3c-4bc3-a971-10953e2299e5-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-zt5k5\" (UID: \"8aec38a2-ea3c-4bc3-a971-10953e2299e5\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:13.151551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:13.151518 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" Oct 31 03:21:13.426713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:13.426630 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5"] Oct 31 03:21:13.429989 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:21:13.429962 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aec38a2_ea3c_4bc3_a971_10953e2299e5.slice/crio-08d647e3a8c74067d8a956d82f8d416d56eb90b0f7e1e1c8ae799b5b91bdd1d3 WatchSource:0}: Error finding container 08d647e3a8c74067d8a956d82f8d416d56eb90b0f7e1e1c8ae799b5b91bdd1d3: Status 404 returned error can't find the container with id 08d647e3a8c74067d8a956d82f8d416d56eb90b0f7e1e1c8ae799b5b91bdd1d3 Oct 31 03:21:14.062282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:14.062248 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" event={"ID":"8aec38a2-ea3c-4bc3-a971-10953e2299e5","Type":"ContainerStarted","Data":"08d647e3a8c74067d8a956d82f8d416d56eb90b0f7e1e1c8ae799b5b91bdd1d3"} Oct 31 03:21:16.694384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:16.694358 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:21:17.075921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:17.075831 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" event={"ID":"8aec38a2-ea3c-4bc3-a971-10953e2299e5","Type":"ContainerStarted","Data":"3eecd0483af6cbc2172feb0b48227eefccc2f813bc1821ef9916b015d6009ae0"} Oct 31 03:21:17.099687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:17.099632 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-zt5k5" podStartSLOduration=1.839895579 podStartE2EDuration="5.09961716s" totalImagesPullingTime="3.259721581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:17.09859017 +0000 UTC m=+301.293113293" watchObservedRunningTime="2026-10-31 03:21:17.09961716 +0000 UTC m=+301.294140322" Oct 31 03:21:24.696184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.696152 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8"] Oct 31 03:21:24.702360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.702339 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:24.706483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.706467 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:21:24.706612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.706586 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:21:24.706717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.706697 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-msbqv\"" Oct 31 03:21:24.716786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.716764 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8"] Oct 31 03:21:24.783633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.783600 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q24gk\" (UniqueName: \"kubernetes.io/projected/b9f94808-efa2-438d-8cf2-20a3c5f93e21-kube-api-access-q24gk\") pod \"kuadrant-operator-controller-manager-678d87766f-654t8\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:24.783782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.783726 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9f94808-efa2-438d-8cf2-20a3c5f93e21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-654t8\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:24.884134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.884100 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9f94808-efa2-438d-8cf2-20a3c5f93e21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-654t8\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:24.884286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.884141 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24gk\" (UniqueName: \"kubernetes.io/projected/b9f94808-efa2-438d-8cf2-20a3c5f93e21-kube-api-access-q24gk\") pod \"kuadrant-operator-controller-manager-678d87766f-654t8\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:24.884526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.884504 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9f94808-efa2-438d-8cf2-20a3c5f93e21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-654t8\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:24.894304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:24.894277 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24gk\" (UniqueName: \"kubernetes.io/projected/b9f94808-efa2-438d-8cf2-20a3c5f93e21-kube-api-access-q24gk\") pod \"kuadrant-operator-controller-manager-678d87766f-654t8\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:25.013792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:25.013701 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:25.287874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:25.287638 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8"] Oct 31 03:21:25.290796 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:21:25.290772 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f94808_efa2_438d_8cf2_20a3c5f93e21.slice/crio-89e9ba7f29b5db4c17ed77501e96545da08e559345a2ab5964b6d6ca498e1ef4 WatchSource:0}: Error finding container 89e9ba7f29b5db4c17ed77501e96545da08e559345a2ab5964b6d6ca498e1ef4: Status 404 returned error can't find the container with id 89e9ba7f29b5db4c17ed77501e96545da08e559345a2ab5964b6d6ca498e1ef4 Oct 31 03:21:26.110195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:26.110155 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" event={"ID":"b9f94808-efa2-438d-8cf2-20a3c5f93e21","Type":"ContainerStarted","Data":"89e9ba7f29b5db4c17ed77501e96545da08e559345a2ab5964b6d6ca498e1ef4"} Oct 31 03:21:30.128097 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:30.128056 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" event={"ID":"b9f94808-efa2-438d-8cf2-20a3c5f93e21","Type":"ContainerStarted","Data":"1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5"} Oct 31 03:21:30.128580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:30.128189 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:30.148726 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:30.148670 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" podStartSLOduration=1.948105127 podStartE2EDuration="6.148654451s" totalImagesPullingTime="4.200549324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:30.146951415 +0000 UTC m=+314.341474535" watchObservedRunningTime="2026-10-31 03:21:30.148654451 +0000 UTC m=+314.343177576" Oct 31 03:21:39.357335 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:21:39.357292 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:21:41.134911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:41.134882 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 03:21:59.559251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.559216 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg"] Oct 31 03:21:59.565950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.565928 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.572015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.571987 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg"] Oct 31 03:21:59.584779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.584745 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fzbgx\"" Oct 31 03:21:59.584779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.584746 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 03:21:59.596260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.596178 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 03:21:59.672451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.672418 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk74b\" (UniqueName: \"kubernetes.io/projected/8cf2112a-59fe-42b2-923d-3bd8e988fa05-kube-api-access-lk74b\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.672625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.672473 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cf2112a-59fe-42b2-923d-3bd8e988fa05-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.672625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.672592 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf2112a-59fe-42b2-923d-3bd8e988fa05-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.773544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.773505 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk74b\" (UniqueName: \"kubernetes.io/projected/8cf2112a-59fe-42b2-923d-3bd8e988fa05-kube-api-access-lk74b\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.773714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.773561 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cf2112a-59fe-42b2-923d-3bd8e988fa05-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.773714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.773600 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf2112a-59fe-42b2-923d-3bd8e988fa05-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.774225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.774161 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cf2112a-59fe-42b2-923d-3bd8e988fa05-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.776148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.776129 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cf2112a-59fe-42b2-923d-3bd8e988fa05-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.782611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.782591 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk74b\" (UniqueName: \"kubernetes.io/projected/8cf2112a-59fe-42b2-923d-3bd8e988fa05-kube-api-access-lk74b\") pod \"kuadrant-console-plugin-66c4fb575d-8xbmg\" (UID: \"8cf2112a-59fe-42b2-923d-3bd8e988fa05\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:21:59.900460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:21:59.900378 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" Oct 31 03:22:00.172089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.172006 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg"] Oct 31 03:22:00.174990 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:22:00.174965 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cf2112a_59fe_42b2_923d_3bd8e988fa05.slice/crio-7a962c8945a2abee45804533f91609e7501d1a141b779accc4078050e03fc9fd WatchSource:0}: Error finding container 7a962c8945a2abee45804533f91609e7501d1a141b779accc4078050e03fc9fd: Status 404 returned error can't find the container with id 7a962c8945a2abee45804533f91609e7501d1a141b779accc4078050e03fc9fd Oct 31 03:22:00.176585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.176562 5520 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:22:00.253597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.253554 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" event={"ID":"8cf2112a-59fe-42b2-923d-3bd8e988fa05","Type":"ContainerStarted","Data":"7a962c8945a2abee45804533f91609e7501d1a141b779accc4078050e03fc9fd"} Oct 31 03:22:00.410640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.410605 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vchh8"] Oct 31 03:22:00.415727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.415706 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" Oct 31 03:22:00.426111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.426091 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b2rkt\"" Oct 31 03:22:00.426433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.426414 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vchh8"] Oct 31 03:22:00.486693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.486656 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpm4\" (UniqueName: \"kubernetes.io/projected/7ad58a8d-f4e3-4043-9141-740aa8594f0f-kube-api-access-ttpm4\") pod \"authorino-56c6c9ddd5-vchh8\" (UID: \"7ad58a8d-f4e3-4043-9141-740aa8594f0f\") " pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" Oct 31 03:22:00.587638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.587603 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpm4\" (UniqueName: \"kubernetes.io/projected/7ad58a8d-f4e3-4043-9141-740aa8594f0f-kube-api-access-ttpm4\") pod \"authorino-56c6c9ddd5-vchh8\" (UID: \"7ad58a8d-f4e3-4043-9141-740aa8594f0f\") " pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" Oct 31 03:22:00.597017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.596996 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpm4\" (UniqueName: \"kubernetes.io/projected/7ad58a8d-f4e3-4043-9141-740aa8594f0f-kube-api-access-ttpm4\") pod \"authorino-56c6c9ddd5-vchh8\" (UID: \"7ad58a8d-f4e3-4043-9141-740aa8594f0f\") " pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" Oct 31 03:22:00.727732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:00.727698 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" Oct 31 03:22:01.004999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:01.004921 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vchh8"] Oct 31 03:22:01.008980 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:22:01.008953 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad58a8d_f4e3_4043_9141_740aa8594f0f.slice/crio-bf1ba7d84be1443eaa1c66d0983d9ebf60c163a69115e31c2c667fbbdde80bf2 WatchSource:0}: Error finding container bf1ba7d84be1443eaa1c66d0983d9ebf60c163a69115e31c2c667fbbdde80bf2: Status 404 returned error can't find the container with id bf1ba7d84be1443eaa1c66d0983d9ebf60c163a69115e31c2c667fbbdde80bf2 Oct 31 03:22:01.259712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:01.259630 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" event={"ID":"7ad58a8d-f4e3-4043-9141-740aa8594f0f","Type":"ContainerStarted","Data":"bf1ba7d84be1443eaa1c66d0983d9ebf60c163a69115e31c2c667fbbdde80bf2"} Oct 31 03:22:05.281341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:05.281304 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" event={"ID":"7ad58a8d-f4e3-4043-9141-740aa8594f0f","Type":"ContainerStarted","Data":"6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236"} Oct 31 03:22:05.327303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:05.327241 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" podStartSLOduration=1.9392454940000001 podStartE2EDuration="5.327222347s" totalImagesPullingTime="3.387976853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:05.327097559 +0000 UTC m=+349.521620705" watchObservedRunningTime="2026-10-31 03:22:05.327222347 +0000 UTC m=+349.521745462" Oct 31 03:22:07.299459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:07.299421 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" event={"ID":"8cf2112a-59fe-42b2-923d-3bd8e988fa05","Type":"ContainerStarted","Data":"6e09d59a79f98c3ec1d6a624566075521deb12609c2c5862039dd1a718fa8b47"} Oct 31 03:22:07.335007 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:07.334946 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8xbmg" podStartSLOduration=2.23610213 podStartE2EDuration="8.334933651s" totalImagesPullingTime="6.098831521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:07.333704028 +0000 UTC m=+351.528227160" watchObservedRunningTime="2026-10-31 03:22:07.334933651 +0000 UTC m=+351.529456772" Oct 31 03:22:08.665273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.665242 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl"] Oct 31 03:22:08.668858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.668835 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:08.677236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.677196 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 03:22:08.677463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.677444 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5bgtf\"" Oct 31 03:22:08.677570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.677522 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 03:22:08.678256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.678235 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 03:22:08.681435 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.681410 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl"] Oct 31 03:22:08.698884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.698857 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s22vp"] Oct 31 03:22:08.702244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.702194 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:08.715763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.715741 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s22vp"] Oct 31 03:22:08.759767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.759736 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr4pr\" (UniqueName: \"kubernetes.io/projected/3f4ca9e0-a291-44dc-9b45-6548f58f9349-kube-api-access-rr4pr\") pod \"llmisvc-controller-manager-869d5dcc5d-wd8tl\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:08.759934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.759783 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1442a889-201b-44a2-ac6a-b9d37d5f81fe-tls-certs\") pod \"model-serving-api-75bd89fdfc-s22vp\" (UID: \"1442a889-201b-44a2-ac6a-b9d37d5f81fe\") " pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:08.759934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.759803 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrxr\" (UniqueName: \"kubernetes.io/projected/1442a889-201b-44a2-ac6a-b9d37d5f81fe-kube-api-access-wwrxr\") pod \"model-serving-api-75bd89fdfc-s22vp\" (UID: \"1442a889-201b-44a2-ac6a-b9d37d5f81fe\") " pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:08.760025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.759951 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f4ca9e0-a291-44dc-9b45-6548f58f9349-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wd8tl\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:08.777564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.777533 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 03:22:08.777698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.777627 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-qjz7r\"" Oct 31 03:22:08.860997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.860968 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1442a889-201b-44a2-ac6a-b9d37d5f81fe-tls-certs\") pod \"model-serving-api-75bd89fdfc-s22vp\" (UID: \"1442a889-201b-44a2-ac6a-b9d37d5f81fe\") " pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:08.860997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.861007 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrxr\" (UniqueName: \"kubernetes.io/projected/1442a889-201b-44a2-ac6a-b9d37d5f81fe-kube-api-access-wwrxr\") pod \"model-serving-api-75bd89fdfc-s22vp\" (UID: \"1442a889-201b-44a2-ac6a-b9d37d5f81fe\") " pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:08.861277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.861075 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f4ca9e0-a291-44dc-9b45-6548f58f9349-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wd8tl\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:08.861277 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:22:08.861122 5520 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Oct 31 03:22:08.861277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.861142 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr4pr\" (UniqueName: \"kubernetes.io/projected/3f4ca9e0-a291-44dc-9b45-6548f58f9349-kube-api-access-rr4pr\") pod \"llmisvc-controller-manager-869d5dcc5d-wd8tl\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:08.861277 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:22:08.861195 5520 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1442a889-201b-44a2-ac6a-b9d37d5f81fe-tls-certs podName:1442a889-201b-44a2-ac6a-b9d37d5f81fe nodeName:}" failed. No retries permitted until 2026-10-31 03:22:09.361179013 +0000 UTC m=+353.555702112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1442a889-201b-44a2-ac6a-b9d37d5f81fe-tls-certs") pod "model-serving-api-75bd89fdfc-s22vp" (UID: "1442a889-201b-44a2-ac6a-b9d37d5f81fe") : secret "model-serving-api-tls" not found Oct 31 03:22:08.863713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.863692 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f4ca9e0-a291-44dc-9b45-6548f58f9349-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-wd8tl\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:08.879057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.879026 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrxr\" (UniqueName: \"kubernetes.io/projected/1442a889-201b-44a2-ac6a-b9d37d5f81fe-kube-api-access-wwrxr\") pod \"model-serving-api-75bd89fdfc-s22vp\" (UID: \"1442a889-201b-44a2-ac6a-b9d37d5f81fe\") " pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:08.879166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.879078 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr4pr\" (UniqueName: \"kubernetes.io/projected/3f4ca9e0-a291-44dc-9b45-6548f58f9349-kube-api-access-rr4pr\") pod \"llmisvc-controller-manager-869d5dcc5d-wd8tl\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:08.982022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:08.981986 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:09.081068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.081036 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8"] Oct 31 03:22:09.085393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.085378 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.101879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.101852 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8"] Oct 31 03:22:09.163060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.163026 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.163263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.163077 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.163263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.163099 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.163263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.163164 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72kt\" (UniqueName: \"kubernetes.io/projected/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-kube-api-access-f72kt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.163263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.163236 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.163407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.163285 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.163407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.163376 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.257272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.257178 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl"] Oct 31 03:22:09.260803 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:22:09.260776 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3f4ca9e0_a291_44dc_9b45_6548f58f9349.slice/crio-8fc90056317edda558ce4e88d848228ad7fdf229c81790e79df20cb42edc0d0a WatchSource:0}: Error finding container 8fc90056317edda558ce4e88d848228ad7fdf229c81790e79df20cb42edc0d0a: Status 404 returned error can't find the container with id 8fc90056317edda558ce4e88d848228ad7fdf229c81790e79df20cb42edc0d0a Oct 31 03:22:09.264059 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264036 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.264153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264077 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.264153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264103 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.264153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264126 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72kt\" (UniqueName: \"kubernetes.io/projected/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-kube-api-access-f72kt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.264338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264172 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.264338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264250 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.264338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264299 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.264916 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.264849 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.266681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.266655 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.266778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.266685 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.266778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.266737 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.266856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.266785 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.271670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.271651 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.272597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.272575 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72kt\" (UniqueName: \"kubernetes.io/projected/6afccb7d-3c0d-4953-9b3b-2f912d6e4b34-kube-api-access-f72kt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-w62v8\" (UID: \"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.307945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.307911 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" event={"ID":"3f4ca9e0-a291-44dc-9b45-6548f58f9349","Type":"ContainerStarted","Data":"8fc90056317edda558ce4e88d848228ad7fdf229c81790e79df20cb42edc0d0a"} Oct 31 03:22:09.357646 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:22:09.357609 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:22:09.365140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.365116 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1442a889-201b-44a2-ac6a-b9d37d5f81fe-tls-certs\") pod \"model-serving-api-75bd89fdfc-s22vp\" (UID: \"1442a889-201b-44a2-ac6a-b9d37d5f81fe\") " pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:09.367582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.367561 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1442a889-201b-44a2-ac6a-b9d37d5f81fe-tls-certs\") pod \"model-serving-api-75bd89fdfc-s22vp\" (UID: \"1442a889-201b-44a2-ac6a-b9d37d5f81fe\") " pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:09.397067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.397037 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:09.615306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.615219 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:09.679832 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.679800 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8"] Oct 31 03:22:09.683126 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:22:09.683100 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afccb7d_3c0d_4953_9b3b_2f912d6e4b34.slice/crio-ced12b69ad7204c087275bf7126325425b8cd256f86926f1288c55ce7e9d0c09 WatchSource:0}: Error finding container ced12b69ad7204c087275bf7126325425b8cd256f86926f1288c55ce7e9d0c09: Status 404 returned error can't find the container with id ced12b69ad7204c087275bf7126325425b8cd256f86926f1288c55ce7e9d0c09 Oct 31 03:22:09.685370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.685338 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:09.685503 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.685396 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:22:09.895546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:09.895467 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-s22vp"] Oct 31 03:22:09.898853 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:22:09.898819 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1442a889_201b_44a2_ac6a_b9d37d5f81fe.slice/crio-00bbe2a56572ae8dc4a488769b47216b692cb0319af6b1898c02a4b414f42474 WatchSource:0}: Error finding container 00bbe2a56572ae8dc4a488769b47216b692cb0319af6b1898c02a4b414f42474: Status 404 returned error can't find the container with id 00bbe2a56572ae8dc4a488769b47216b692cb0319af6b1898c02a4b414f42474 Oct 31 03:22:10.318622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:10.318577 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" event={"ID":"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34","Type":"ContainerStarted","Data":"9777514709f0bc349a2a1128f455c50c841f5c21d32c3334c2f8b4bf7383b44a"} Oct 31 03:22:10.318922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:10.318897 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" event={"ID":"6afccb7d-3c0d-4953-9b3b-2f912d6e4b34","Type":"ContainerStarted","Data":"ced12b69ad7204c087275bf7126325425b8cd256f86926f1288c55ce7e9d0c09"} Oct 31 03:22:10.319072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:10.319050 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:10.320338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:10.320312 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" event={"ID":"1442a889-201b-44a2-ac6a-b9d37d5f81fe","Type":"ContainerStarted","Data":"00bbe2a56572ae8dc4a488769b47216b692cb0319af6b1898c02a4b414f42474"} Oct 31 03:22:10.320749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:10.320724 5520 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-w62v8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:22:10.320848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:10.320771 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" podUID="6afccb7d-3c0d-4953-9b3b-2f912d6e4b34" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:22:10.385991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:10.385941 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" podStartSLOduration=1.385926287 podStartE2EDuration="1.385926287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:10.382589159 +0000 UTC m=+354.577112280" watchObservedRunningTime="2026-10-31 03:22:10.385926287 +0000 UTC m=+354.580449408" Oct 31 03:22:11.326223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.326166 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-w62v8" Oct 31 03:22:11.376511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.376483 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b"] Oct 31 03:22:11.376718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.376686 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" podUID="e9c8bb12-78d0-4cb7-8841-26b91d1a3099" containerName="discovery" containerID="cri-o://6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414" gracePeriod=30 Oct 31 03:22:11.652356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.652329 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:22:11.687030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.686998 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-kube-api-access-m6b2c\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-kube-api-access-m6b2c\") pod \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " Oct 31 03:22:11.687235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.687054 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-kubeconfig\") pod \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " Oct 31 03:22:11.687235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.687092 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-token\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-token\") pod \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " Oct 31 03:22:11.687235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.687154 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-dns-cert\") pod \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " Oct 31 03:22:11.687235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.687190 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-ca-configmap\") pod \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " Oct 31 03:22:11.687495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.687254 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-cacerts\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-cacerts\") pod \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " Oct 31 03:22:11.687495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.687278 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-local-certs\") pod \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\" (UID: \"e9c8bb12-78d0-4cb7-8841-26b91d1a3099\") " Oct 31 03:22:11.688255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.688185 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-ca-configmap" pod "e9c8bb12-78d0-4cb7-8841-26b91d1a3099" (UID: "e9c8bb12-78d0-4cb7-8841-26b91d1a3099"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:11.690358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.690328 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-kube-api-access-m6b2c" pod "e9c8bb12-78d0-4cb7-8841-26b91d1a3099" (UID: "e9c8bb12-78d0-4cb7-8841-26b91d1a3099"). InnerVolumeSpecName "kube-api-access-m6b2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:11.690635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.690607 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-dns-cert" pod "e9c8bb12-78d0-4cb7-8841-26b91d1a3099" (UID: "e9c8bb12-78d0-4cb7-8841-26b91d1a3099"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:11.690719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.690655 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-kubeconfig" pod "e9c8bb12-78d0-4cb7-8841-26b91d1a3099" (UID: "e9c8bb12-78d0-4cb7-8841-26b91d1a3099"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:11.690878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.690861 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-local-certs" pod "e9c8bb12-78d0-4cb7-8841-26b91d1a3099" (UID: "e9c8bb12-78d0-4cb7-8841-26b91d1a3099"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:22:11.691032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.691008 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-token" pod "e9c8bb12-78d0-4cb7-8841-26b91d1a3099" (UID: "e9c8bb12-78d0-4cb7-8841-26b91d1a3099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:11.691170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.691150 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-cacerts" pod "e9c8bb12-78d0-4cb7-8841-26b91d1a3099" (UID: "e9c8bb12-78d0-4cb7-8841-26b91d1a3099"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:11.788438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.788413 5520 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-cacerts\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:11.788536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.788443 5520 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-local-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:11.788536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.788457 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6b2c\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-kube-api-access-m6b2c\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:11.788536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.788468 5520 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-kubeconfig\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:11.788536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.788476 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:11.788536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.788484 5520 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-dns-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:11.788536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:11.788493 5520 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9c8bb12-78d0-4cb7-8841-26b91d1a3099-istio-csr-ca-configmap\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:12.330961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.330927 5520 generic.go:356] "Generic (PLEG): container finished" podID="e9c8bb12-78d0-4cb7-8841-26b91d1a3099" containerID="6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414" exitCode=0 Oct 31 03:22:12.331416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.330981 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" Oct 31 03:22:12.331416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.331018 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" event={"ID":"e9c8bb12-78d0-4cb7-8841-26b91d1a3099","Type":"ContainerDied","Data":"6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414"} Oct 31 03:22:12.331416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.331079 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b" event={"ID":"e9c8bb12-78d0-4cb7-8841-26b91d1a3099","Type":"ContainerDied","Data":"b4b88b601a793242e548d692dede826e04f7f72d413a08ccfd58b3647d267473"} Oct 31 03:22:12.331416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.331105 5520 scope.go:122] "RemoveContainer" containerID="6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414" Oct 31 03:22:12.335330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.335253 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" event={"ID":"1442a889-201b-44a2-ac6a-b9d37d5f81fe","Type":"ContainerStarted","Data":"9042fe0bf2fc3660b591f6363e2bc44830c802b9b962e99081404d0cbb433107"} Oct 31 03:22:12.341798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.341780 5520 scope.go:122] "RemoveContainer" containerID="6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414" Oct 31 03:22:12.342081 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:22:12.342060 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414\": container with ID starting with 6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414 not found: ID does not exist" containerID="6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414" Oct 31 03:22:12.342140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.342091 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414"} err="failed to get container status \"6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414\": rpc error: code = NotFound desc = could not find container \"6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414\": container with ID starting with 6e51c66ff6d39c9203dd8cdb79e18ef35d68e9b1f7f80080ac993c4d349a3414 not found: ID does not exist" Oct 31 03:22:12.351362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.351241 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b"] Oct 31 03:22:12.355310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.355276 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-62d4b"] Oct 31 03:22:12.369313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:12.369266 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" podStartSLOduration=2.531380246 podStartE2EDuration="4.369249638s" totalImagesPullingTime="1.837869392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:12.368948832 +0000 UTC m=+356.563471954" watchObservedRunningTime="2026-10-31 03:22:12.369249638 +0000 UTC m=+356.563772760" Oct 31 03:22:13.341265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:13.341230 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" event={"ID":"3f4ca9e0-a291-44dc-9b45-6548f58f9349","Type":"ContainerStarted","Data":"a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8"} Oct 31 03:22:13.341700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:13.341327 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:22:13.342607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:13.342589 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:13.361756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:13.361696 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" podStartSLOduration=2.2857047169999998 podStartE2EDuration="5.361675567s" totalImagesPullingTime="3.07597085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:13.361258753 +0000 UTC m=+357.555781888" watchObservedRunningTime="2026-10-31 03:22:13.361675567 +0000 UTC m=+357.556198689" Oct 31 03:22:13.914678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:13.914648 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c8bb12-78d0-4cb7-8841-26b91d1a3099" path="/var/lib/kubelet/pods/e9c8bb12-78d0-4cb7-8841-26b91d1a3099/volumes" Oct 31 03:22:24.352249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:24.352192 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-s22vp" Oct 31 03:22:39.357659 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:22:39.357576 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:22:44.349684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:22:44.349656 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:23:09.357496 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:23:09.357454 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:23:24.470746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.470710 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n48wm"] Oct 31 03:23:24.471253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.471033 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e9c8bb12-78d0-4cb7-8841-26b91d1a3099" containerName="discovery" Oct 31 03:23:24.471253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.471044 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c8bb12-78d0-4cb7-8841-26b91d1a3099" containerName="discovery" Oct 31 03:23:24.471253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.471123 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c8bb12-78d0-4cb7-8841-26b91d1a3099" containerName="discovery" Oct 31 03:23:24.474179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.474161 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:24.489783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.485512 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 03:23:24.489783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.487017 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n48wm"] Oct 31 03:23:24.633630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.633598 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/85257fab-90e7-4297-9598-2129e3afd82b-tls-cert\") pod \"authorino-6889cbd8c4-n48wm\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:24.633806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.633728 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrsk\" (UniqueName: \"kubernetes.io/projected/85257fab-90e7-4297-9598-2129e3afd82b-kube-api-access-2rrsk\") pod \"authorino-6889cbd8c4-n48wm\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:24.696184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.696154 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n48wm"] Oct 31 03:23:24.696391 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:23:24.696371 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2rrsk tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-n48wm" podUID="85257fab-90e7-4297-9598-2129e3afd82b" Oct 31 03:23:24.734398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.734321 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/85257fab-90e7-4297-9598-2129e3afd82b-tls-cert\") pod \"authorino-6889cbd8c4-n48wm\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:24.734537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.734441 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrsk\" (UniqueName: \"kubernetes.io/projected/85257fab-90e7-4297-9598-2129e3afd82b-kube-api-access-2rrsk\") pod \"authorino-6889cbd8c4-n48wm\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:24.737174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.737151 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/85257fab-90e7-4297-9598-2129e3afd82b-tls-cert\") pod \"authorino-6889cbd8c4-n48wm\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:24.742309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:24.742290 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrsk\" (UniqueName: \"kubernetes.io/projected/85257fab-90e7-4297-9598-2129e3afd82b-kube-api-access-2rrsk\") pod \"authorino-6889cbd8c4-n48wm\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:25.635861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.635822 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:25.643036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.643013 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:25.744439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.744402 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85257fab-90e7-4297-9598-2129e3afd82b-tls-cert\" (UniqueName: \"kubernetes.io/secret/85257fab-90e7-4297-9598-2129e3afd82b-tls-cert\") pod \"85257fab-90e7-4297-9598-2129e3afd82b\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " Oct 31 03:23:25.744904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.744883 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85257fab-90e7-4297-9598-2129e3afd82b-kube-api-access-2rrsk\" (UniqueName: \"kubernetes.io/projected/85257fab-90e7-4297-9598-2129e3afd82b-kube-api-access-2rrsk\") pod \"85257fab-90e7-4297-9598-2129e3afd82b\" (UID: \"85257fab-90e7-4297-9598-2129e3afd82b\") " Oct 31 03:23:25.746942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.746912 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85257fab-90e7-4297-9598-2129e3afd82b-tls-cert" pod "85257fab-90e7-4297-9598-2129e3afd82b" (UID: "85257fab-90e7-4297-9598-2129e3afd82b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:25.747044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.747020 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85257fab-90e7-4297-9598-2129e3afd82b-kube-api-access-2rrsk" pod "85257fab-90e7-4297-9598-2129e3afd82b" (UID: "85257fab-90e7-4297-9598-2129e3afd82b"). InnerVolumeSpecName "kube-api-access-2rrsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:25.846437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.846405 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rrsk\" (UniqueName: \"kubernetes.io/projected/85257fab-90e7-4297-9598-2129e3afd82b-kube-api-access-2rrsk\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:25.846437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:25.846432 5520 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/85257fab-90e7-4297-9598-2129e3afd82b-tls-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:26.640084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:26.640055 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n48wm" Oct 31 03:23:26.669914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:26.669870 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n48wm"] Oct 31 03:23:26.675750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:26.675719 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n48wm"] Oct 31 03:23:27.741441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:27.741404 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vchh8"] Oct 31 03:23:27.741977 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:27.741563 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" podUID="7ad58a8d-f4e3-4043-9141-740aa8594f0f" containerName="authorino" containerID="cri-o://6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236" gracePeriod=30 Oct 31 03:23:27.915350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:27.915099 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85257fab-90e7-4297-9598-2129e3afd82b" path="/var/lib/kubelet/pods/85257fab-90e7-4297-9598-2129e3afd82b/volumes" Oct 31 03:23:27.982167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:27.982143 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" Oct 31 03:23:28.064059 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.063969 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad58a8d-f4e3-4043-9141-740aa8594f0f-kube-api-access-ttpm4\" (UniqueName: \"kubernetes.io/projected/7ad58a8d-f4e3-4043-9141-740aa8594f0f-kube-api-access-ttpm4\") pod \"7ad58a8d-f4e3-4043-9141-740aa8594f0f\" (UID: \"7ad58a8d-f4e3-4043-9141-740aa8594f0f\") " Oct 31 03:23:28.066374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.066344 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad58a8d-f4e3-4043-9141-740aa8594f0f-kube-api-access-ttpm4" pod "7ad58a8d-f4e3-4043-9141-740aa8594f0f" (UID: "7ad58a8d-f4e3-4043-9141-740aa8594f0f"). InnerVolumeSpecName "kube-api-access-ttpm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:28.165336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.165302 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttpm4\" (UniqueName: \"kubernetes.io/projected/7ad58a8d-f4e3-4043-9141-740aa8594f0f-kube-api-access-ttpm4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:28.605355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.605318 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9"] Oct 31 03:23:28.605650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.605639 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7ad58a8d-f4e3-4043-9141-740aa8594f0f" containerName="authorino" Oct 31 03:23:28.605695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.605651 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad58a8d-f4e3-4043-9141-740aa8594f0f" containerName="authorino" Oct 31 03:23:28.605732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.605717 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad58a8d-f4e3-4043-9141-740aa8594f0f" containerName="authorino" Oct 31 03:23:28.608546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.608527 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:28.620984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.620961 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9"] Oct 31 03:23:28.650185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.650157 5520 generic.go:356] "Generic (PLEG): container finished" podID="7ad58a8d-f4e3-4043-9141-740aa8594f0f" containerID="6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236" exitCode=0 Oct 31 03:23:28.650338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.650194 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" Oct 31 03:23:28.650338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.650195 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" event={"ID":"7ad58a8d-f4e3-4043-9141-740aa8594f0f","Type":"ContainerDied","Data":"6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236"} Oct 31 03:23:28.650338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.650264 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-vchh8" event={"ID":"7ad58a8d-f4e3-4043-9141-740aa8594f0f","Type":"ContainerDied","Data":"bf1ba7d84be1443eaa1c66d0983d9ebf60c163a69115e31c2c667fbbdde80bf2"} Oct 31 03:23:28.650338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.650278 5520 scope.go:122] "RemoveContainer" containerID="6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236" Oct 31 03:23:28.658551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.658523 5520 scope.go:122] "RemoveContainer" containerID="6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236" Oct 31 03:23:28.658823 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:23:28.658802 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236\": container with ID starting with 6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236 not found: ID does not exist" containerID="6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236" Oct 31 03:23:28.658895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.658837 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236"} err="failed to get container status \"6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236\": rpc error: code = NotFound desc = could not find container \"6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236\": container with ID starting with 6121e8b9c0c04f99e3c41cf9aa8a84037642079e3250e5365aaef627d3242236 not found: ID does not exist" Oct 31 03:23:28.669469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.669445 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vchh8"] Oct 31 03:23:28.669596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.669580 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ce577b-f8f3-40d6-ba23-6b57cbfcea65-cert\") pod \"llmisvc-controller-manager-6466c554d8-ntrp9\" (UID: \"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:28.669638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.669617 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwdm\" (UniqueName: \"kubernetes.io/projected/b4ce577b-f8f3-40d6-ba23-6b57cbfcea65-kube-api-access-2gwdm\") pod \"llmisvc-controller-manager-6466c554d8-ntrp9\" (UID: \"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:28.673482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.673463 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-vchh8"] Oct 31 03:23:28.770896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.770864 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ce577b-f8f3-40d6-ba23-6b57cbfcea65-cert\") pod \"llmisvc-controller-manager-6466c554d8-ntrp9\" (UID: \"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:28.771274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.770926 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwdm\" (UniqueName: \"kubernetes.io/projected/b4ce577b-f8f3-40d6-ba23-6b57cbfcea65-kube-api-access-2gwdm\") pod \"llmisvc-controller-manager-6466c554d8-ntrp9\" (UID: \"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:28.773398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.773380 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4ce577b-f8f3-40d6-ba23-6b57cbfcea65-cert\") pod \"llmisvc-controller-manager-6466c554d8-ntrp9\" (UID: \"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:28.778597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.778572 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwdm\" (UniqueName: \"kubernetes.io/projected/b4ce577b-f8f3-40d6-ba23-6b57cbfcea65-kube-api-access-2gwdm\") pod \"llmisvc-controller-manager-6466c554d8-ntrp9\" (UID: \"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:28.921031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:28.920954 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:29.191930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:29.191902 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9"] Oct 31 03:23:29.195018 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:23:29.194976 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ce577b_f8f3_40d6_ba23_6b57cbfcea65.slice/crio-c64f953c3b20972fd59c919fa680e77cc37a6c6432fb532d5a74f078d62228b3 WatchSource:0}: Error finding container c64f953c3b20972fd59c919fa680e77cc37a6c6432fb532d5a74f078d62228b3: Status 404 returned error can't find the container with id c64f953c3b20972fd59c919fa680e77cc37a6c6432fb532d5a74f078d62228b3 Oct 31 03:23:29.656010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:29.655981 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" event={"ID":"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65","Type":"ContainerStarted","Data":"c64f953c3b20972fd59c919fa680e77cc37a6c6432fb532d5a74f078d62228b3"} Oct 31 03:23:29.919617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:29.919544 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad58a8d-f4e3-4043-9141-740aa8594f0f" path="/var/lib/kubelet/pods/7ad58a8d-f4e3-4043-9141-740aa8594f0f/volumes" Oct 31 03:23:30.662711 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:30.662677 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" event={"ID":"b4ce577b-f8f3-40d6-ba23-6b57cbfcea65","Type":"ContainerStarted","Data":"6a6b9eff7dce7a01e74c189ed4e234f5a24f7e98e14903c6da56cc7ab8b9c8ba"} Oct 31 03:23:30.662885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:30.662810 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:23:30.679318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:23:30.679269 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" podStartSLOduration=2.266046042 podStartE2EDuration="2.679255998s" totalImagesPullingTime="413.209956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:30.679114228 +0000 UTC m=+434.873637368" watchObservedRunningTime="2026-10-31 03:23:30.679255998 +0000 UTC m=+434.873779118" Oct 31 03:23:39.357137 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:23:39.357099 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:24:01.669174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:01.669101 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntrp9" Oct 31 03:24:01.711532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:01.711499 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl"] Oct 31 03:24:01.711766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:01.711739 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" podUID="3f4ca9e0-a291-44dc-9b45-6548f58f9349" containerName="manager" containerID="cri-o://a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8" gracePeriod=30 Oct 31 03:24:01.951508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:01.951485 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:24:02.052141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.052099 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4ca9e0-a291-44dc-9b45-6548f58f9349-kube-api-access-rr4pr\" (UniqueName: \"kubernetes.io/projected/3f4ca9e0-a291-44dc-9b45-6548f58f9349-kube-api-access-rr4pr\") pod \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " Oct 31 03:24:02.052344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.052157 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4ca9e0-a291-44dc-9b45-6548f58f9349-cert\" (UniqueName: \"kubernetes.io/secret/3f4ca9e0-a291-44dc-9b45-6548f58f9349-cert\") pod \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\" (UID: \"3f4ca9e0-a291-44dc-9b45-6548f58f9349\") " Oct 31 03:24:02.054483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.054449 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4ca9e0-a291-44dc-9b45-6548f58f9349-cert" pod "3f4ca9e0-a291-44dc-9b45-6548f58f9349" (UID: "3f4ca9e0-a291-44dc-9b45-6548f58f9349"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:02.054580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.054525 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4ca9e0-a291-44dc-9b45-6548f58f9349-kube-api-access-rr4pr" pod "3f4ca9e0-a291-44dc-9b45-6548f58f9349" (UID: "3f4ca9e0-a291-44dc-9b45-6548f58f9349"). InnerVolumeSpecName "kube-api-access-rr4pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:02.153491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.153456 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr4pr\" (UniqueName: \"kubernetes.io/projected/3f4ca9e0-a291-44dc-9b45-6548f58f9349-kube-api-access-rr4pr\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:02.153491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.153489 5520 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f4ca9e0-a291-44dc-9b45-6548f58f9349-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:02.793680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.793643 5520 generic.go:356] "Generic (PLEG): container finished" podID="3f4ca9e0-a291-44dc-9b45-6548f58f9349" containerID="a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8" exitCode=0 Oct 31 03:24:02.794089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.793693 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" Oct 31 03:24:02.794089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.793713 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" event={"ID":"3f4ca9e0-a291-44dc-9b45-6548f58f9349","Type":"ContainerDied","Data":"a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8"} Oct 31 03:24:02.794089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.793763 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl" event={"ID":"3f4ca9e0-a291-44dc-9b45-6548f58f9349","Type":"ContainerDied","Data":"8fc90056317edda558ce4e88d848228ad7fdf229c81790e79df20cb42edc0d0a"} Oct 31 03:24:02.794089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.793796 5520 scope.go:122] "RemoveContainer" containerID="a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8" Oct 31 03:24:02.802699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.802683 5520 scope.go:122] "RemoveContainer" containerID="a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8" Oct 31 03:24:02.802950 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:24:02.802928 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8\": container with ID starting with a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8 not found: ID does not exist" containerID="a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8" Oct 31 03:24:02.802989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.802960 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8"} err="failed to get container status \"a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8\": rpc error: code = NotFound desc = could not find container \"a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8\": container with ID starting with a5259b2b17d75701fb916c2f31dc2de6b3de53f8ff42a24103e0ff6f1b9db2d8 not found: ID does not exist" Oct 31 03:24:02.813878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.813856 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl"] Oct 31 03:24:02.822363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:02.822343 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-wd8tl"] Oct 31 03:24:03.914518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:03.914488 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4ca9e0-a291-44dc-9b45-6548f58f9349" path="/var/lib/kubelet/pods/3f4ca9e0-a291-44dc-9b45-6548f58f9349/volumes" Oct 31 03:24:09.357442 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:24:09.357401 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:24:39.357650 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:24:39.357607 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:24:41.513566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.513537 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2"] Oct 31 03:24:41.513943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.513882 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3f4ca9e0-a291-44dc-9b45-6548f58f9349" containerName="manager" Oct 31 03:24:41.513943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.513893 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4ca9e0-a291-44dc-9b45-6548f58f9349" containerName="manager" Oct 31 03:24:41.514012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.513954 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4ca9e0-a291-44dc-9b45-6548f58f9349" containerName="manager" Oct 31 03:24:41.516746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.516725 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.535239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.535072 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2"] Oct 31 03:24:41.602019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.601983 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tw4cw\"" Oct 31 03:24:41.606441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606411 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fhf9\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-kube-api-access-9fhf9\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606459 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a2997-9eea-48fc-87a9-3e255f11746f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606524 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606545 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606574 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606711 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606751 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606784 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.606807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.606800 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a2997-9eea-48fc-87a9-3e255f11746f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708040 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708080 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a2997-9eea-48fc-87a9-3e255f11746f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708244 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhf9\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-kube-api-access-9fhf9\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708303 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a2997-9eea-48fc-87a9-3e255f11746f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708335 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708361 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708417 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708479 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708507 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708528 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708733 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708809 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.708896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.708866 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.709153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.709128 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a2997-9eea-48fc-87a9-3e255f11746f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.710733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.710711 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.710929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.710912 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a2997-9eea-48fc-87a9-3e255f11746f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.716160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.716136 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.716358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.716336 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fhf9\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-kube-api-access-9fhf9\") pod \"maas-default-gateway-openshift-default-769df7846c-77vz2\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:41.831661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:41.831575 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:42.110283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:42.110102 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2"] Oct 31 03:24:42.113272 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:24:42.113243 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026a2997_9eea_48fc_87a9_3e255f11746f.slice/crio-50362b0b3f2a5d017cec5a893808f22c6bf15945e41bdbf82c84a112263346ca WatchSource:0}: Error finding container 50362b0b3f2a5d017cec5a893808f22c6bf15945e41bdbf82c84a112263346ca: Status 404 returned error can't find the container with id 50362b0b3f2a5d017cec5a893808f22c6bf15945e41bdbf82c84a112263346ca Oct 31 03:24:42.963039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:42.962997 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" event={"ID":"026a2997-9eea-48fc-87a9-3e255f11746f","Type":"ContainerStarted","Data":"50362b0b3f2a5d017cec5a893808f22c6bf15945e41bdbf82c84a112263346ca"} Oct 31 03:24:46.121231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:46.121152 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:24:46.121802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:46.121277 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:24:46.121802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:46.121339 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:24:47.000809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:47.000754 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" event={"ID":"026a2997-9eea-48fc-87a9-3e255f11746f","Type":"ContainerStarted","Data":"edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c"} Oct 31 03:24:47.021294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:47.021232 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podStartSLOduration=2.016697633 podStartE2EDuration="6.021216502s" totalImagesPullingTime="4.004518869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:47.019476318 +0000 UTC m=+511.213999450" watchObservedRunningTime="2026-10-31 03:24:47.021216502 +0000 UTC m=+511.215739614" Oct 31 03:24:47.832775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:47.832738 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:47.834098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:47.834074 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:24:47.834242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:47.834121 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 03:24:48.832677 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:48.832641 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:24:48.832857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:48.832709 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 03:24:49.832588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:49.832547 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:24:49.832761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:49.832621 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 03:24:50.832651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:50.832607 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:24:50.833129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:50.832667 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 03:24:51.832626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:51.832587 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:24:51.832804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:51.832661 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 03:24:52.831960 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:52.831920 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 31 03:24:52.832139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:52.831984 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 31 03:24:54.832471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:54.832425 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:24:54.832995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:54.832506 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:24:55.874435 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:55.874395 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:24:55.874922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:55.874456 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:24:56.378492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.378453 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:56.378992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.378967 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:56.379790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.379764 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:24:56.591506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.591477 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx"] Oct 31 03:24:56.594965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.594943 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.613510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.613485 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx"] Oct 31 03:24:56.675449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675368 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675406 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675429 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675487 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675558 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739ca5c0-6636-4c97-9236-04a5af2564d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675584 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675604 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vw6\" (UniqueName: \"kubernetes.io/projected/739ca5c0-6636-4c97-9236-04a5af2564d6-kube-api-access-59vw6\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675701 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.675811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.675738 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.776710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.776674 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.776871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.776729 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.776871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.776786 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.776871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.776809 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.776871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.776827 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.776927 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.776980 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739ca5c0-6636-4c97-9236-04a5af2564d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.777019 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.777052 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vw6\" (UniqueName: \"kubernetes.io/projected/739ca5c0-6636-4c97-9236-04a5af2564d6-kube-api-access-59vw6\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.777128 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.777156 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.777464 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.777706 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.777772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.777711 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739ca5c0-6636-4c97-9236-04a5af2564d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.779309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.779290 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.779419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.779402 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.785998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.785970 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739ca5c0-6636-4c97-9236-04a5af2564d6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.786586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.786563 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vw6\" (UniqueName: \"kubernetes.io/projected/739ca5c0-6636-4c97-9236-04a5af2564d6-kube-api-access-59vw6\") pod \"maas-default-gateway-openshift-default-769df7846c-5wkxx\" (UID: \"739ca5c0-6636-4c97-9236-04a5af2564d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:56.909514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:56.909480 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:57.187107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:57.187074 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx"] Oct 31 03:24:57.190090 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:24:57.190051 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739ca5c0_6636_4c97_9236_04a5af2564d6.slice/crio-b4c7755b93f891e282e3d9f6f7d394eda49cc509a20c21c59c42f6b7849f47a8 WatchSource:0}: Error finding container b4c7755b93f891e282e3d9f6f7d394eda49cc509a20c21c59c42f6b7849f47a8: Status 404 returned error can't find the container with id b4c7755b93f891e282e3d9f6f7d394eda49cc509a20c21c59c42f6b7849f47a8 Oct 31 03:24:57.193240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:57.193186 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:24:57.193310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:57.193290 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:24:57.193347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:57.193332 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:24:58.055635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:58.055596 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" event={"ID":"739ca5c0-6636-4c97-9236-04a5af2564d6","Type":"ContainerStarted","Data":"15bff5252c12d917a5197d8c3f80695326b916b4cc8990012d28bc2421ca4804"} Oct 31 03:24:58.056155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:58.055652 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" event={"ID":"739ca5c0-6636-4c97-9236-04a5af2564d6","Type":"ContainerStarted","Data":"b4c7755b93f891e282e3d9f6f7d394eda49cc509a20c21c59c42f6b7849f47a8"} Oct 31 03:24:58.084779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:58.084175 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podStartSLOduration=2.08415329 podStartE2EDuration="2.08415329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:58.080034439 +0000 UTC m=+522.274557560" watchObservedRunningTime="2026-10-31 03:24:58.08415329 +0000 UTC m=+522.278676390" Oct 31 03:24:58.910195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:58.910158 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:24:58.911638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:58.911613 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:24:58.911747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:58.911660 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 03:24:59.909908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:59.909861 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:24:59.910390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:24:59.909940 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 03:25:00.910696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:00.910654 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:25:00.911122 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:00.910724 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 03:25:01.909870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:01.909834 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:25:01.910035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:01.909894 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 03:25:02.910071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:02.910024 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:25:02.910550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:02.910095 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 03:25:03.910180 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:03.910146 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 03:25:03.910690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:03.910215 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 03:25:05.328522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:05.328493 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:25:05.329882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:05.328913 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:25:05.329882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:05.329620 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" Oct 31 03:25:09.356765 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:25:09.356729 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:25:11.596530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.596499 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl"] Oct 31 03:25:11.598843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.598824 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.616862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.616830 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl"] Oct 31 03:25:11.713749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.713710 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.713749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.713752 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.713956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.713778 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.713956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.713898 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.713956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.713922 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.713956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.713942 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.714085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.714037 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.714085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.714070 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtcxl\" (UniqueName: \"kubernetes.io/projected/91077eb7-9c1a-4363-bdbf-e65fac8395d8-kube-api-access-vtcxl\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.714169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.714156 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815478 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815512 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815532 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815651 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815695 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815720 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815744 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815808 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.815867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815863 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.816131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815881 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtcxl\" (UniqueName: \"kubernetes.io/projected/91077eb7-9c1a-4363-bdbf-e65fac8395d8-kube-api-access-vtcxl\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.816131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815927 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.816131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.815984 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.816131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.816057 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.816494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.816475 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.818136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.818114 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.818442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.818424 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.828148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.828124 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91077eb7-9c1a-4363-bdbf-e65fac8395d8-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.828268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.828247 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtcxl\" (UniqueName: \"kubernetes.io/projected/91077eb7-9c1a-4363-bdbf-e65fac8395d8-kube-api-access-vtcxl\") pod \"maas-default-gateway-openshift-default-769df7846c-f6mgl\" (UID: \"91077eb7-9c1a-4363-bdbf-e65fac8395d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:11.909894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:11.909803 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:12.187370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:12.183474 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl"] Oct 31 03:25:12.189744 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:25:12.189712 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91077eb7_9c1a_4363_bdbf_e65fac8395d8.slice/crio-6dbc0b72a9b4f183ceba4c78a523c751c941e48908284dbd4288bb9d9f77e57d WatchSource:0}: Error finding container 6dbc0b72a9b4f183ceba4c78a523c751c941e48908284dbd4288bb9d9f77e57d: Status 404 returned error can't find the container with id 6dbc0b72a9b4f183ceba4c78a523c751c941e48908284dbd4288bb9d9f77e57d Oct 31 03:25:12.192819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:12.192789 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:25:12.192911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:12.192851 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:25:12.192911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:12.192878 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:25:13.136998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:13.136963 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" event={"ID":"91077eb7-9c1a-4363-bdbf-e65fac8395d8","Type":"ContainerStarted","Data":"3e6818e2a3e935f4df7b7ac28951f98b5efcdd984f48dede79cce03ea10bee01"} Oct 31 03:25:13.137419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:13.137015 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" event={"ID":"91077eb7-9c1a-4363-bdbf-e65fac8395d8","Type":"ContainerStarted","Data":"6dbc0b72a9b4f183ceba4c78a523c751c941e48908284dbd4288bb9d9f77e57d"} Oct 31 03:25:13.158659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:13.158620 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podStartSLOduration=2.1586088119999998 podStartE2EDuration="2.158608812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:13.158033343 +0000 UTC m=+537.352556466" watchObservedRunningTime="2026-10-31 03:25:13.158608812 +0000 UTC m=+537.353131927" Oct 31 03:25:13.911396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:13.911359 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:25:13.911565 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:13.911433 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:25:13.914600 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:13.914577 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:14.911071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:14.911033 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:25:14.911540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:14.911088 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:25:15.911106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:15.911064 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:25:15.911618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:15.911135 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:25:16.910412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:16.910378 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:25:16.910694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:16.910447 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:25:17.910848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:17.910805 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:25:17.911290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:17.910879 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:25:18.910470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:18.910423 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 03:25:18.910643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:18.910505 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 03:25:20.371843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:20.371809 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:20.372360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:20.372183 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:20.373104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:20.373089 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" Oct 31 03:25:39.357447 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:25:39.357366 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:25:49.430741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.430709 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf"] Oct 31 03:25:49.434419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.434402 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.440683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.440652 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7f3159-openshift-default-dockercfg-4mdls\"" Oct 31 03:25:49.460564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.460530 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf"] Oct 31 03:25:49.511152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511110 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511153 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511290 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511327 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511407 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d435696-c420-4d28-84bb-550440e03636-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511452 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511470 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d435696-c420-4d28-84bb-550440e03636-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511492 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.511693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.511564 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkqg\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-kube-api-access-sqkqg\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.615673 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.615631 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.615914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.615897 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.616022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.616010 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d435696-c420-4d28-84bb-550440e03636-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.616136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.616125 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.616239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.616226 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d435696-c420-4d28-84bb-550440e03636-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.616344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.616329 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.616431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.616420 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkqg\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-kube-api-access-sqkqg\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.616560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.616550 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.616639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.616629 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.617416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.617394 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.617798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.617777 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.618657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.618628 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d435696-c420-4d28-84bb-550440e03636-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.620070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.619025 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.620228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.619266 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.624653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.624617 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.625833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.625811 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d435696-c420-4d28-84bb-550440e03636-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.636707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.636683 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.636873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.636734 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkqg\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-kube-api-access-sqkqg\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:49.748670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:49.748593 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:50.090781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.090650 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf"] Oct 31 03:25:50.092985 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:25:50.092948 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d435696_c420_4d28_84bb_550440e03636.slice/crio-3dab127d2bb97a40c28c987e3e803ddec6a169148c6464ed1c1d0d4974d6ef53 WatchSource:0}: Error finding container 3dab127d2bb97a40c28c987e3e803ddec6a169148c6464ed1c1d0d4974d6ef53: Status 404 returned error can't find the container with id 3dab127d2bb97a40c28c987e3e803ddec6a169148c6464ed1c1d0d4974d6ef53 Oct 31 03:25:50.095041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.095013 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:25:50.095111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.095086 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:25:50.095154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.095113 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:25:50.300612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.300573 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" event={"ID":"3d435696-c420-4d28-84bb-550440e03636","Type":"ContainerStarted","Data":"5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22"} Oct 31 03:25:50.300799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.300646 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" event={"ID":"3d435696-c420-4d28-84bb-550440e03636","Type":"ContainerStarted","Data":"3dab127d2bb97a40c28c987e3e803ddec6a169148c6464ed1c1d0d4974d6ef53"} Oct 31 03:25:50.336942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.336862 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podStartSLOduration=1.336839903 podStartE2EDuration="1.336839903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:50.335850991 +0000 UTC m=+574.530374140" watchObservedRunningTime="2026-10-31 03:25:50.336839903 +0000 UTC m=+574.531363031" Oct 31 03:25:50.753234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:50.750323 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:51.754418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:51.754391 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:52.310413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:52.310386 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:52.311343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:52.311322 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:25:58.329002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.328972 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5"] Oct 31 03:25:58.332413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.332382 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.341185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.341161 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-bcc7a4\"" Oct 31 03:25:58.341368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.341289 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-bcc7a4-dockercfg-mzz4z\"" Oct 31 03:25:58.354393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.354363 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5"] Oct 31 03:25:58.519683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.519638 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntdv\" (UniqueName: \"kubernetes.io/projected/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-kube-api-access-mntdv\") pod \"payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.519871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.519697 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.621312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.621196 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntdv\" (UniqueName: \"kubernetes.io/projected/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-kube-api-access-mntdv\") pod \"payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.621312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.621268 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.622259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.622234 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.642098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.642059 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntdv\" (UniqueName: \"kubernetes.io/projected/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-kube-api-access-mntdv\") pod \"payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.646850 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.646826 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:25:58.960174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:58.960151 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5"] Oct 31 03:25:58.962302 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:25:58.962274 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ba03ee_18ea_42dc_a954_8d6b3ab652cd.slice/crio-286c350a89a4aa997da37c2dcebeaf64c3473215599af31bd23df3a2eec466e5 WatchSource:0}: Error finding container 286c350a89a4aa997da37c2dcebeaf64c3473215599af31bd23df3a2eec466e5: Status 404 returned error can't find the container with id 286c350a89a4aa997da37c2dcebeaf64c3473215599af31bd23df3a2eec466e5 Oct 31 03:25:59.346941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:25:59.346869 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" event={"ID":"57ba03ee-18ea-42dc-a954-8d6b3ab652cd","Type":"ContainerStarted","Data":"286c350a89a4aa997da37c2dcebeaf64c3473215599af31bd23df3a2eec466e5"} Oct 31 03:26:00.444843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.444807 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf"] Oct 31 03:26:00.453636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.453612 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:00.460230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.460176 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-6c6050-dockercfg-4m9mr\"" Oct 31 03:26:00.462033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.461710 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf"] Oct 31 03:26:00.504292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.504256 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6c6050\"" Oct 31 03:26:00.551934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.551886 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkldc\" (UniqueName: \"kubernetes.io/projected/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-kube-api-access-wkldc\") pod \"payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:00.552119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.552053 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:00.652771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.652735 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:00.652771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.652774 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkldc\" (UniqueName: \"kubernetes.io/projected/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-kube-api-access-wkldc\") pod \"payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:00.653572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.653546 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:00.665182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.665157 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkldc\" (UniqueName: \"kubernetes.io/projected/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-kube-api-access-wkldc\") pod \"payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:00.771859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:00.771781 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:01.051603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.051526 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl"] Oct 31 03:26:01.056985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.056959 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.065945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.065914 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-dda5cb\"" Oct 31 03:26:01.068001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.067924 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl"] Oct 31 03:26:01.099947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.099853 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf"] Oct 31 03:26:01.103467 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:01.103433 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bed3b9_0e41_42dd_a0af_86a9526f3b82.slice/crio-92622aad593d07da663e68b45a6878d771279a37b11ee19c5ba62e2e2a6125e4 WatchSource:0}: Error finding container 92622aad593d07da663e68b45a6878d771279a37b11ee19c5ba62e2e2a6125e4: Status 404 returned error can't find the container with id 92622aad593d07da663e68b45a6878d771279a37b11ee19c5ba62e2e2a6125e4 Oct 31 03:26:01.157344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.157311 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9ft\" (UniqueName: \"kubernetes.io/projected/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-kube-api-access-tv9ft\") pod \"payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.157519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.157362 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.258379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.258342 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.258538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.258455 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9ft\" (UniqueName: \"kubernetes.io/projected/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-kube-api-access-tv9ft\") pod \"payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.259336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.259309 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.267086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.267057 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9ft\" (UniqueName: \"kubernetes.io/projected/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-kube-api-access-tv9ft\") pod \"payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.360689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.360602 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" event={"ID":"b4bed3b9-0e41-42dd-a0af-86a9526f3b82","Type":"ContainerStarted","Data":"92622aad593d07da663e68b45a6878d771279a37b11ee19c5ba62e2e2a6125e4"} Oct 31 03:26:01.376840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.376811 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:01.400638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.400607 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h"] Oct 31 03:26:01.410400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.409745 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.418908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.418342 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h"] Oct 31 03:26:01.420145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.419647 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbwgh\"" Oct 31 03:26:01.420145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.419743 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:26:01.420145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.419878 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:26:01.420145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.419951 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:26:01.420145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.420009 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f20dc4-serving-cert\"" Oct 31 03:26:01.460511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.460178 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqtnd\" (UniqueName: \"kubernetes.io/projected/92500990-fcda-4877-8297-c8d53af11174-kube-api-access-xqtnd\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.460511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.460242 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92500990-fcda-4877-8297-c8d53af11174-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.460511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.460298 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-metrics-certs\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.460511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.460351 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.561346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.561310 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqtnd\" (UniqueName: \"kubernetes.io/projected/92500990-fcda-4877-8297-c8d53af11174-kube-api-access-xqtnd\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.561724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.561352 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92500990-fcda-4877-8297-c8d53af11174-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.561724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.561392 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-metrics-certs\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.561724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.561442 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.562497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.562403 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92500990-fcda-4877-8297-c8d53af11174-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.564401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.564358 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.564777 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.564745 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-metrics-certs\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.578086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.578023 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqtnd\" (UniqueName: \"kubernetes.io/projected/92500990-fcda-4877-8297-c8d53af11174-kube-api-access-xqtnd\") pod \"maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:01.694326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.694293 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl"] Oct 31 03:26:01.699353 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:01.699318 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc1a4d2_a402_470e_9dd2_31a9fb33be0d.slice/crio-8266e503eace3aa7fa608596708e952c4baf44acec6cd6abc3ab1b52621cd4e5 WatchSource:0}: Error finding container 8266e503eace3aa7fa608596708e952c4baf44acec6cd6abc3ab1b52621cd4e5: Status 404 returned error can't find the container with id 8266e503eace3aa7fa608596708e952c4baf44acec6cd6abc3ab1b52621cd4e5 Oct 31 03:26:01.729802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:01.729770 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:02.030718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.030681 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h"] Oct 31 03:26:02.035267 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:02.035230 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92500990_fcda_4877_8297_c8d53af11174.slice/crio-c7c280da0d6489d169cd44a2e137b70271d59aac6f8cc3efbff6bb4151de29d5 WatchSource:0}: Error finding container c7c280da0d6489d169cd44a2e137b70271d59aac6f8cc3efbff6bb4151de29d5: Status 404 returned error can't find the container with id c7c280da0d6489d169cd44a2e137b70271d59aac6f8cc3efbff6bb4151de29d5 Oct 31 03:26:02.366457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.366373 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" event={"ID":"92500990-fcda-4877-8297-c8d53af11174","Type":"ContainerStarted","Data":"c7c280da0d6489d169cd44a2e137b70271d59aac6f8cc3efbff6bb4151de29d5"} Oct 31 03:26:02.367908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.367872 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" event={"ID":"57ba03ee-18ea-42dc-a954-8d6b3ab652cd","Type":"ContainerStarted","Data":"ebd41b63858b5c2597d18c87fc8d100ac469dd53e3164947bc53960cd0576f32"} Oct 31 03:26:02.368041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.367970 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:26:02.369578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.369542 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" event={"ID":"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d","Type":"ContainerStarted","Data":"b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b"} Oct 31 03:26:02.369681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.369603 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" event={"ID":"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d","Type":"ContainerStarted","Data":"8266e503eace3aa7fa608596708e952c4baf44acec6cd6abc3ab1b52621cd4e5"} Oct 31 03:26:02.369681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.369635 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:02.371022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.371000 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" event={"ID":"b4bed3b9-0e41-42dd-a0af-86a9526f3b82","Type":"ContainerStarted","Data":"578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12"} Oct 31 03:26:02.371131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.371117 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:02.392775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.392713 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" podStartSLOduration=1.737887879 podStartE2EDuration="4.392694536s" totalImagesPullingTime="2.654806657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:02.391938527 +0000 UTC m=+586.586461663" watchObservedRunningTime="2026-10-31 03:26:02.392694536 +0000 UTC m=+586.587217660" Oct 31 03:26:02.421939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.421880 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" podStartSLOduration=1.774925336 podStartE2EDuration="2.421864918s" totalImagesPullingTime="646.939582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:02.42085038 +0000 UTC m=+586.615373501" watchObservedRunningTime="2026-10-31 03:26:02.421864918 +0000 UTC m=+586.616388040" Oct 31 03:26:02.443303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:02.443259 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" podStartSLOduration=1.44324509 podStartE2EDuration="1.44324509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:02.442389137 +0000 UTC m=+586.636912286" watchObservedRunningTime="2026-10-31 03:26:02.44324509 +0000 UTC m=+586.637768211" Oct 31 03:26:04.908474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.905702 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x"] Oct 31 03:26:04.933225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.922130 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:04.933225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.932136 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x"] Oct 31 03:26:04.950312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.950284 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-6c6050-serving-cert\"" Oct 31 03:26:04.994518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.994483 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b474775-af3d-4761-ae90-066a630dbdc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:04.994705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.994536 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:04.994705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.994675 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-metrics-certs\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:04.994843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:04.994761 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl25c\" (UniqueName: \"kubernetes.io/projected/4b474775-af3d-4761-ae90-066a630dbdc8-kube-api-access-jl25c\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.095657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.095630 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b474775-af3d-4761-ae90-066a630dbdc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.095770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.095677 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.095770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.095720 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-metrics-certs\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.095770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.095752 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl25c\" (UniqueName: \"kubernetes.io/projected/4b474775-af3d-4761-ae90-066a630dbdc8-kube-api-access-jl25c\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.096641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.096616 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b474775-af3d-4761-ae90-066a630dbdc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.098301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.098269 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.099144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.099117 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-metrics-certs\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.113042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.113012 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl25c\" (UniqueName: \"kubernetes.io/projected/4b474775-af3d-4761-ae90-066a630dbdc8-kube-api-access-jl25c\") pod \"maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.257605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.257571 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:05.392195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.392162 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" event={"ID":"92500990-fcda-4877-8297-c8d53af11174","Type":"ContainerStarted","Data":"2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea"} Oct 31 03:26:05.392414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.392255 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:05.541716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.541584 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" podStartSLOduration=1.506926939 podStartE2EDuration="4.541567916s" totalImagesPullingTime="3.034640977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:05.432726545 +0000 UTC m=+589.627249666" watchObservedRunningTime="2026-10-31 03:26:05.541567916 +0000 UTC m=+589.736091036" Oct 31 03:26:05.541788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:05.541724 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x"] Oct 31 03:26:05.544425 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:05.544396 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b474775_af3d_4761_ae90_066a630dbdc8.slice/crio-80f33380a30c44761d0fb4597727464b6058346cbfec8a021ed68cd533b20a70 WatchSource:0}: Error finding container 80f33380a30c44761d0fb4597727464b6058346cbfec8a021ed68cd533b20a70: Status 404 returned error can't find the container with id 80f33380a30c44761d0fb4597727464b6058346cbfec8a021ed68cd533b20a70 Oct 31 03:26:06.399596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:06.399500 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" event={"ID":"4b474775-af3d-4761-ae90-066a630dbdc8","Type":"ContainerStarted","Data":"e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b"} Oct 31 03:26:06.399596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:06.399567 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" event={"ID":"4b474775-af3d-4761-ae90-066a630dbdc8","Type":"ContainerStarted","Data":"80f33380a30c44761d0fb4597727464b6058346cbfec8a021ed68cd533b20a70"} Oct 31 03:26:06.400011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:06.399599 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:06.423012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:06.422960 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" podStartSLOduration=1.933872853 podStartE2EDuration="2.422943166s" totalImagesPullingTime="489.070313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:06.4226894 +0000 UTC m=+590.617212535" watchObservedRunningTime="2026-10-31 03:26:06.422943166 +0000 UTC m=+590.617466287" Oct 31 03:26:09.357714 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:26:09.357673 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:26:11.426303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:11.422407 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:26:12.416456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:12.416423 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:26:13.398749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:13.398720 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:26:13.399194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:13.398802 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 03:26:13.399194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:13.398964 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:26:15.110369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.110113 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf"] Oct 31 03:26:15.114731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.114708 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.130402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.130369 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"simulator-w4-dda5cb-kserve-self-signed-certs\"" Oct 31 03:26:15.130549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.130506 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"default-dockercfg-2qpsx\"" Oct 31 03:26:15.131251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.130601 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"kube-root-ca.crt\"" Oct 31 03:26:15.131251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.130680 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"openshift-service-ca.crt\"" Oct 31 03:26:15.131251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.130961 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf"] Oct 31 03:26:15.257597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.257561 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tmp-dir\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.257597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.257599 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-model-cache\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.257853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.257687 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-dshm\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.257853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.257724 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-home\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.257853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.257845 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxl6\" (UniqueName: \"kubernetes.io/projected/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-kube-api-access-9fxl6\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.258017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.257894 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tls-certs\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359279 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-home\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359369 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxl6\" (UniqueName: \"kubernetes.io/projected/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-kube-api-access-9fxl6\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359431 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tls-certs\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359465 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tmp-dir\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359493 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-model-cache\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359647 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-dshm\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359758 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-home\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359809 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-model-cache\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.359953 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.359899 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tmp-dir\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.361983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.361917 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-dshm\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.362149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.362130 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tls-certs\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.372222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.371808 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxl6\" (UniqueName: \"kubernetes.io/projected/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-kube-api-access-9fxl6\") pod \"simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.432376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.432337 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:15.769871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:15.769831 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf"] Oct 31 03:26:16.511153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:16.511107 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" event={"ID":"b1ba2dce-da62-4cea-bfce-7c77a1c6121d","Type":"ContainerStarted","Data":"363ef84774cf61cd206ae19ac402de9f27e1072074a82e3e58defbee53f66220"} Oct 31 03:26:18.523533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:18.523488 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" event={"ID":"b1ba2dce-da62-4cea-bfce-7c77a1c6121d","Type":"ContainerStarted","Data":"060c9b0a7716add8af1731b281f69ced8edca1c6bad3946aeef16e4a1f1870b1"} Oct 31 03:26:18.572380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:18.572316 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podStartSLOduration=1.891922868 podStartE2EDuration="3.572297313s" totalImagesPullingTime="1.680374445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:18.569725179 +0000 UTC m=+602.764248296" watchObservedRunningTime="2026-10-31 03:26:18.572297313 +0000 UTC m=+602.766820434" Oct 31 03:26:19.433486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:19.433450 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:19.531788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:19.531703 5520 generic.go:356] "Generic (PLEG): container finished" podID="3d435696-c420-4d28-84bb-550440e03636" containerID="5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22" exitCode=0 Oct 31 03:26:19.532188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:19.531782 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" event={"ID":"3d435696-c420-4d28-84bb-550440e03636","Type":"ContainerDied","Data":"5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22"} Oct 31 03:26:19.532394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:19.532377 5520 scope.go:122] "RemoveContainer" containerID="5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22" Oct 31 03:26:19.533821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:19.533787 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:26:19.533930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:19.533876 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:26:19.533930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:19.533922 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:26:20.540946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:20.540907 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" event={"ID":"3d435696-c420-4d28-84bb-550440e03636","Type":"ContainerStarted","Data":"758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c"} Oct 31 03:26:20.754594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:20.754504 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:26:20.756416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:20.755990 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:26:20.756416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:20.756062 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:26:21.748961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:21.748925 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:26:21.749427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:21.748984 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:26:22.310489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.310455 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:26:22.749212 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.749179 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:26:22.749615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.749257 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:26:22.962112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.962065 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj"] Oct 31 03:26:22.974230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.969436 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:22.980141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.980105 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"kube-root-ca.crt\"" Oct 31 03:26:22.987757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.987733 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"openshift-service-ca.crt\"" Oct 31 03:26:22.998835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.988039 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"default-dockercfg-bvtsj\"" Oct 31 03:26:22.999121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:22.998682 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"simulator-w5-bcc7a4-kserve-self-signed-certs\"" Oct 31 03:26:23.007104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.007032 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj"] Oct 31 03:26:23.050956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.050916 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-dshm\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.051173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.051121 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tls-certs\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.051274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.051221 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-model-cache\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.051274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.051254 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-home\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.051384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.051298 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzn2b\" (UniqueName: \"kubernetes.io/projected/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-kube-api-access-pzn2b\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.051384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.051340 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tmp-dir\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.156779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.156738 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-dshm\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.157618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.157594 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tls-certs\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.157828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.157810 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-model-cache\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.157950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.157936 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-home\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.158087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.158069 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzn2b\" (UniqueName: \"kubernetes.io/projected/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-kube-api-access-pzn2b\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.158219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.158187 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tmp-dir\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.158684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.158660 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-model-cache\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.158829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.158686 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tmp-dir\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.158927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.158890 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-home\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.167479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.167452 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-dshm\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.183223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.180584 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzn2b\" (UniqueName: \"kubernetes.io/projected/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-kube-api-access-pzn2b\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.183223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.181104 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tls-certs\") pod \"simulator-w5-bcc7a4-kserve-7db9449744-vgddj\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.321009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.320924 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:23.677281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.673898 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj"] Oct 31 03:26:23.704804 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:23.704766 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f6ddb2f_0908_47ff_8eca_76242df4ae8d.slice/crio-9bbf16162e57487a66618257f6d456e2ff6402c524646dea82902c0594a65aaf WatchSource:0}: Error finding container 9bbf16162e57487a66618257f6d456e2ff6402c524646dea82902c0594a65aaf: Status 404 returned error can't find the container with id 9bbf16162e57487a66618257f6d456e2ff6402c524646dea82902c0594a65aaf Oct 31 03:26:23.783328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.783268 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:26:23.783853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:23.783361 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:26:24.564149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:24.564115 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" event={"ID":"1f6ddb2f-0908-47ff-8eca-76242df4ae8d","Type":"ContainerStarted","Data":"a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982"} Oct 31 03:26:24.564149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:24.564155 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" event={"ID":"1f6ddb2f-0908-47ff-8eca-76242df4ae8d","Type":"ContainerStarted","Data":"9bbf16162e57487a66618257f6d456e2ff6402c524646dea82902c0594a65aaf"} Oct 31 03:26:24.599265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:24.598950 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podStartSLOduration=2.598934807 podStartE2EDuration="2.598934807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:24.598263972 +0000 UTC m=+608.792787093" watchObservedRunningTime="2026-10-31 03:26:24.598934807 +0000 UTC m=+608.793457927" Oct 31 03:26:24.748971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:24.748940 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 31 03:26:24.749152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:24.748991 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 31 03:26:25.321770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:25.321724 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:25.432909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:25.432875 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:25.445529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:25.445501 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:25.587714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:25.587627 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:26:25.756304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:25.756253 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:26:25.757582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:25.757550 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 03:26:33.321339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.321296 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:33.340859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.340826 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:33.648470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.648400 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:26:33.663623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.663595 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n"] Oct 31 03:26:33.666754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.666730 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.693398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.693324 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n"] Oct 31 03:26:33.693398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.693340 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"default-dockercfg-b78xs\"" Oct 31 03:26:33.693729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.693696 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"simulator-w6-7f3159-kserve-self-signed-certs\"" Oct 31 03:26:33.694264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.694243 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"openshift-service-ca.crt\"" Oct 31 03:26:33.694389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.694367 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"kube-root-ca.crt\"" Oct 31 03:26:33.811646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.811615 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-dshm\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.811908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.811666 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxxgd\" (UniqueName: \"kubernetes.io/projected/b391e679-e308-499e-8b85-022626eeecde-kube-api-access-sxxgd\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.811908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.811823 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-model-cache\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.811908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.811898 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b391e679-e308-499e-8b85-022626eeecde-tls-certs\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.812087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.812028 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-home\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.812148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.812093 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-tmp-dir\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.914714 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-home\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.914774 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-tmp-dir\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.914821 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-dshm\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.914853 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxxgd\" (UniqueName: \"kubernetes.io/projected/b391e679-e308-499e-8b85-022626eeecde-kube-api-access-sxxgd\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.914954 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-model-cache\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.915018 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b391e679-e308-499e-8b85-022626eeecde-tls-certs\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.916246 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-model-cache\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.916511 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-home\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.923276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.916729 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-tmp-dir\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.937317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.937239 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-dshm\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.945153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.945120 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b391e679-e308-499e-8b85-022626eeecde-tls-certs\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.950258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.945600 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxxgd\" (UniqueName: \"kubernetes.io/projected/b391e679-e308-499e-8b85-022626eeecde-kube-api-access-sxxgd\") pod \"simulator-w6-7f3159-kserve-6b696459b4-b5n9n\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:33.978699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:33.978664 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:34.282228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:34.282173 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n"] Oct 31 03:26:34.284989 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:34.284962 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb391e679_e308_499e_8b85_022626eeecde.slice/crio-dbe03aca846e3f0261574d3d52f10ae38674a55918889574375cad1a14f70d32 WatchSource:0}: Error finding container dbe03aca846e3f0261574d3d52f10ae38674a55918889574375cad1a14f70d32: Status 404 returned error can't find the container with id dbe03aca846e3f0261574d3d52f10ae38674a55918889574375cad1a14f70d32 Oct 31 03:26:34.640483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:34.640375 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" event={"ID":"b391e679-e308-499e-8b85-022626eeecde","Type":"ContainerStarted","Data":"1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d"} Oct 31 03:26:34.640483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:34.640427 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" event={"ID":"b391e679-e308-499e-8b85-022626eeecde","Type":"ContainerStarted","Data":"dbe03aca846e3f0261574d3d52f10ae38674a55918889574375cad1a14f70d32"} Oct 31 03:26:34.662782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:34.662719 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podStartSLOduration=1.662699073 podStartE2EDuration="1.662699073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:34.66025201 +0000 UTC m=+618.854775159" watchObservedRunningTime="2026-10-31 03:26:34.662699073 +0000 UTC m=+618.857222198" Oct 31 03:26:34.979027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:34.978995 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:37.483053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.483017 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k"] Oct 31 03:26:37.487150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.487120 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.497240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.497181 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"default-dockercfg-9w2c5\"" Oct 31 03:26:37.497686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.497663 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"simulator-w1-6c6050-kserve-self-signed-certs\"" Oct 31 03:26:37.498167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.498147 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"openshift-service-ca.crt\"" Oct 31 03:26:37.498303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.498275 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"kube-root-ca.crt\"" Oct 31 03:26:37.508238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.504637 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k"] Oct 31 03:26:37.657650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.657613 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tls-certs\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.657956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.657932 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-model-cache\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.658241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.658221 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tmp-dir\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.658394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.658375 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-home\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.658540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.658523 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwzb8\" (UniqueName: \"kubernetes.io/projected/f184c11a-b290-49a2-bab7-0e3bad0dbddd-kube-api-access-rwzb8\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.658663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.658649 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-dshm\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759324 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-dshm\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759412 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tls-certs\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759450 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-model-cache\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759505 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tmp-dir\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759526 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-home\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759547 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwzb8\" (UniqueName: \"kubernetes.io/projected/f184c11a-b290-49a2-bab7-0e3bad0dbddd-kube-api-access-rwzb8\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759900 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tmp-dir\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.759924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759910 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-model-cache\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.760084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.759945 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-home\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.762253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.762230 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-dshm\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.762522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.762505 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tls-certs\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.768401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.768378 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwzb8\" (UniqueName: \"kubernetes.io/projected/f184c11a-b290-49a2-bab7-0e3bad0dbddd-kube-api-access-rwzb8\") pod \"simulator-w1-6c6050-kserve-79856b8cfb-6mf6k\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:37.801954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:37.801913 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:38.078783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:38.078755 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k"] Oct 31 03:26:38.081499 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:38.081465 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf184c11a_b290_49a2_bab7_0e3bad0dbddd.slice/crio-f45c5ab6f0ea8f52a16c73f6b36b3333dff146bc47659c1e58af91944086d876 WatchSource:0}: Error finding container f45c5ab6f0ea8f52a16c73f6b36b3333dff146bc47659c1e58af91944086d876: Status 404 returned error can't find the container with id f45c5ab6f0ea8f52a16c73f6b36b3333dff146bc47659c1e58af91944086d876 Oct 31 03:26:38.691931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:38.691893 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" event={"ID":"f184c11a-b290-49a2-bab7-0e3bad0dbddd","Type":"ContainerStarted","Data":"5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c"} Oct 31 03:26:38.692463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:38.691940 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" event={"ID":"f184c11a-b290-49a2-bab7-0e3bad0dbddd","Type":"ContainerStarted","Data":"f45c5ab6f0ea8f52a16c73f6b36b3333dff146bc47659c1e58af91944086d876"} Oct 31 03:26:38.715095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:38.713703 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podStartSLOduration=1.71368482 podStartE2EDuration="1.71368482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:38.712926686 +0000 UTC m=+622.907449818" watchObservedRunningTime="2026-10-31 03:26:38.71368482 +0000 UTC m=+622.908207942" Oct 31 03:26:38.802011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:38.801973 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:39.358447 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:26:39.358401 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:26:40.491249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.491216 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5"] Oct 31 03:26:40.495166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.495133 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.500491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.500441 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"openshift-service-ca.crt\"" Oct 31 03:26:40.500633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.500501 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"kube-root-ca.crt\"" Oct 31 03:26:40.500633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.500561 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"default-dockercfg-5n7tw\"" Oct 31 03:26:40.501466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.501444 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"simulator-w2-f20dc4-kserve-self-signed-certs\"" Oct 31 03:26:40.563569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.563499 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5"] Oct 31 03:26:40.587626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.587567 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-home\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.587829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.587638 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b642464-78dd-415c-9a8a-35622d1abcbf-tls-certs\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.587829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.587777 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-dshm\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.587829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.587818 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4mnz\" (UniqueName: \"kubernetes.io/projected/6b642464-78dd-415c-9a8a-35622d1abcbf-kube-api-access-x4mnz\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.588002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.587866 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-model-cache\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.588002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.587926 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-tmp-dir\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.688558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.688515 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-dshm\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.688766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.688564 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4mnz\" (UniqueName: \"kubernetes.io/projected/6b642464-78dd-415c-9a8a-35622d1abcbf-kube-api-access-x4mnz\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.688766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.688611 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-model-cache\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.688766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.688639 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-tmp-dir\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.688766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.688736 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-home\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.688981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.688775 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b642464-78dd-415c-9a8a-35622d1abcbf-tls-certs\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.689076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.689050 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-model-cache\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.689259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.689180 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-tmp-dir\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.689334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.689279 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-home\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.691417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.691389 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b642464-78dd-415c-9a8a-35622d1abcbf-tls-certs\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.691525 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.691470 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-dshm\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.706743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.706712 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4mnz\" (UniqueName: \"kubernetes.io/projected/6b642464-78dd-415c-9a8a-35622d1abcbf-kube-api-access-x4mnz\") pod \"simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:40.862056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:40.861969 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:41.143947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:41.143923 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5"] Oct 31 03:26:41.146069 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:41.146038 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b642464_78dd_415c_9a8a_35622d1abcbf.slice/crio-d5ac0994210c82a15584765cdc69c8fdb5f6da585cb38bd0586a2bcdef71f65f WatchSource:0}: Error finding container d5ac0994210c82a15584765cdc69c8fdb5f6da585cb38bd0586a2bcdef71f65f: Status 404 returned error can't find the container with id d5ac0994210c82a15584765cdc69c8fdb5f6da585cb38bd0586a2bcdef71f65f Oct 31 03:26:41.736276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:41.736238 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" event={"ID":"6b642464-78dd-415c-9a8a-35622d1abcbf","Type":"ContainerStarted","Data":"97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05"} Oct 31 03:26:41.736658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:41.736288 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" event={"ID":"6b642464-78dd-415c-9a8a-35622d1abcbf","Type":"ContainerStarted","Data":"d5ac0994210c82a15584765cdc69c8fdb5f6da585cb38bd0586a2bcdef71f65f"} Oct 31 03:26:41.759535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:41.759479 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podStartSLOduration=1.759464215 podStartE2EDuration="1.759464215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:41.757369581 +0000 UTC m=+625.951892702" watchObservedRunningTime="2026-10-31 03:26:41.759464215 +0000 UTC m=+625.953987335" Oct 31 03:26:41.863103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:41.863065 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:43.985225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:43.980528 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:44.027259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:44.027184 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:44.763685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:44.763652 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:26:47.802971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:47.802934 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:47.817021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:47.816992 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:48.407085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.407053 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98"] Oct 31 03:26:48.411256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.411228 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.420624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.420596 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"kube-root-ca.crt\"" Oct 31 03:26:48.420779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.420713 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"openshift-service-ca.crt\"" Oct 31 03:26:48.422541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.421564 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"default-dockercfg-qfn76\"" Oct 31 03:26:48.422541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.421659 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"premium-w0-d86a77-kserve-self-signed-certs\"" Oct 31 03:26:48.423954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.423931 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98"] Oct 31 03:26:48.472708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.472661 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-dshm\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.472911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.472802 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-model-cache\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.472911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.472864 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-home\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.473038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.473005 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78562ae3-0ada-42c7-917f-52c0152b890a-tls-certs\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.473141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.473116 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25mhc\" (UniqueName: \"kubernetes.io/projected/78562ae3-0ada-42c7-917f-52c0152b890a-kube-api-access-25mhc\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.473187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.473160 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-tmp-dir\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.574242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.574177 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-model-cache\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.574691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.574666 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-model-cache\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.574792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.574722 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-home\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.574847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.574791 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78562ae3-0ada-42c7-917f-52c0152b890a-tls-certs\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.574902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.574847 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25mhc\" (UniqueName: \"kubernetes.io/projected/78562ae3-0ada-42c7-917f-52c0152b890a-kube-api-access-25mhc\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.574902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.574873 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-tmp-dir\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.575025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.574937 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-dshm\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.575657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.575626 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-home\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.576178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.576154 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-tmp-dir\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.578556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.578532 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78562ae3-0ada-42c7-917f-52c0152b890a-tls-certs\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.578746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.578727 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-dshm\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.584046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.584020 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25mhc\" (UniqueName: \"kubernetes.io/projected/78562ae3-0ada-42c7-917f-52c0152b890a-kube-api-access-25mhc\") pod \"premium-w0-d86a77-kserve-5496ccf66c-r7m98\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.727887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.727853 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:48.797232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:48.796470 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:26:49.004979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:49.004951 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98"] Oct 31 03:26:49.007479 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:49.007450 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78562ae3_0ada_42c7_917f_52c0152b890a.slice/crio-ce85d064437387d8925ce9f914b939752f86ad3db7239e0278f54f032e3c94cb WatchSource:0}: Error finding container ce85d064437387d8925ce9f914b939752f86ad3db7239e0278f54f032e3c94cb: Status 404 returned error can't find the container with id ce85d064437387d8925ce9f914b939752f86ad3db7239e0278f54f032e3c94cb Oct 31 03:26:49.780083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:49.780048 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" event={"ID":"78562ae3-0ada-42c7-917f-52c0152b890a","Type":"ContainerStarted","Data":"f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9"} Oct 31 03:26:49.780359 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:49.780099 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" event={"ID":"78562ae3-0ada-42c7-917f-52c0152b890a","Type":"ContainerStarted","Data":"ce85d064437387d8925ce9f914b939752f86ad3db7239e0278f54f032e3c94cb"} Oct 31 03:26:49.813987 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:49.813937 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podStartSLOduration=1.813923989 podStartE2EDuration="1.813923989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:49.812553803 +0000 UTC m=+634.007076946" watchObservedRunningTime="2026-10-31 03:26:49.813923989 +0000 UTC m=+634.008447109" Oct 31 03:26:50.728100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:50.728066 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:50.862834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:50.862801 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:50.879541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:50.879508 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:51.802704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:51.802673 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:26:58.691166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.691076 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh"] Oct 31 03:26:58.695807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.695785 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.703492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.703465 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"premium-w6-7f3159-kserve-self-signed-certs\"" Oct 31 03:26:58.705009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.704957 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh"] Oct 31 03:26:58.728394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.728366 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:58.743864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.743835 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:58.804215 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.804159 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-model-cache\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.804401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.804239 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tmp-dir\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.804401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.804274 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tls-certs\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.804492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.804450 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-dshm\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.804492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.804479 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-home\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.804571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.804537 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c57d\" (UniqueName: \"kubernetes.io/projected/8b2833b3-ae04-427f-812c-ed7174eb6d6a-kube-api-access-2c57d\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.881238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.873567 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:26:58.909316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.908790 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-model-cache\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.909316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.908853 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tmp-dir\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.909316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.908887 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tls-certs\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.909316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.908971 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-dshm\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.909316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.909000 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-home\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.909316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.909057 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c57d\" (UniqueName: \"kubernetes.io/projected/8b2833b3-ae04-427f-812c-ed7174eb6d6a-kube-api-access-2c57d\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.913231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.909784 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-model-cache\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.913231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.910050 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tmp-dir\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.914091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.914063 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-home\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.915065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.915007 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-dshm\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.935914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.935851 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tls-certs\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:58.938581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:58.938456 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c57d\" (UniqueName: \"kubernetes.io/projected/8b2833b3-ae04-427f-812c-ed7174eb6d6a-kube-api-access-2c57d\") pod \"premium-w6-7f3159-kserve-8544497f85-bdgrh\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:59.023284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:59.019548 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:26:59.341397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:59.341247 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh"] Oct 31 03:26:59.344491 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:26:59.344463 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b2833b3_ae04_427f_812c_ed7174eb6d6a.slice/crio-a85e37017213e0e8da81625c4ebe2a6f56ddfeec12fb0354377b983deda5dd18 WatchSource:0}: Error finding container a85e37017213e0e8da81625c4ebe2a6f56ddfeec12fb0354377b983deda5dd18: Status 404 returned error can't find the container with id a85e37017213e0e8da81625c4ebe2a6f56ddfeec12fb0354377b983deda5dd18 Oct 31 03:26:59.845632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:59.845554 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" event={"ID":"8b2833b3-ae04-427f-812c-ed7174eb6d6a","Type":"ContainerStarted","Data":"e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e"} Oct 31 03:26:59.845632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:59.845603 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" event={"ID":"8b2833b3-ae04-427f-812c-ed7174eb6d6a","Type":"ContainerStarted","Data":"a85e37017213e0e8da81625c4ebe2a6f56ddfeec12fb0354377b983deda5dd18"} Oct 31 03:26:59.899475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:26:59.899402 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podStartSLOduration=1.8993806279999998 podStartE2EDuration="1.899380628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:59.895870958 +0000 UTC m=+644.090394182" watchObservedRunningTime="2026-10-31 03:26:59.899380628 +0000 UTC m=+644.093903750" Oct 31 03:27:00.027786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:00.023350 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:27:03.802475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.800636 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52"] Oct 31 03:27:03.804742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.804719 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.813109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.813076 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52"] Oct 31 03:27:03.813948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.813925 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"premium-w1-6c6050-kserve-self-signed-certs\"" Oct 31 03:27:03.881813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.881777 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-tmp-dir\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.882010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.881823 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e4171-ca09-4c7f-afbe-08a2fa33a736-tls-certs\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.882010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.881909 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg67q\" (UniqueName: \"kubernetes.io/projected/195e4171-ca09-4c7f-afbe-08a2fa33a736-kube-api-access-sg67q\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.882150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.882039 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-model-cache\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.882190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.882146 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-home\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.882190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.882183 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-dshm\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.984238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.983825 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-tmp-dir\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.984238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.983882 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e4171-ca09-4c7f-afbe-08a2fa33a736-tls-certs\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.984238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.983927 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg67q\" (UniqueName: \"kubernetes.io/projected/195e4171-ca09-4c7f-afbe-08a2fa33a736-kube-api-access-sg67q\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.984238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.983981 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-model-cache\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.984238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.984034 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-home\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.984238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.984057 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-dshm\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.988236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.986735 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-model-cache\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.988236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.986778 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-home\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:03.988236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:03.987006 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-tmp-dir\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:04.013224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.007707 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-dshm\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:04.013224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.009948 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg67q\" (UniqueName: \"kubernetes.io/projected/195e4171-ca09-4c7f-afbe-08a2fa33a736-kube-api-access-sg67q\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:04.013224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.011116 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e4171-ca09-4c7f-afbe-08a2fa33a736-tls-certs\") pod \"premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:04.131531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.131438 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:04.415263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.415229 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52"] Oct 31 03:27:04.417328 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:27:04.417302 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod195e4171_ca09_4c7f_afbe_08a2fa33a736.slice/crio-5e5d29eaa6ac787936c8f9745f6ca0691bf5c89c10fcd166f4716ed0434955c9 WatchSource:0}: Error finding container 5e5d29eaa6ac787936c8f9745f6ca0691bf5c89c10fcd166f4716ed0434955c9: Status 404 returned error can't find the container with id 5e5d29eaa6ac787936c8f9745f6ca0691bf5c89c10fcd166f4716ed0434955c9 Oct 31 03:27:04.874301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.874254 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" event={"ID":"195e4171-ca09-4c7f-afbe-08a2fa33a736","Type":"ContainerStarted","Data":"cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4"} Oct 31 03:27:04.874301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.874303 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" event={"ID":"195e4171-ca09-4c7f-afbe-08a2fa33a736","Type":"ContainerStarted","Data":"5e5d29eaa6ac787936c8f9745f6ca0691bf5c89c10fcd166f4716ed0434955c9"} Oct 31 03:27:04.893759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:04.893710 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podStartSLOduration=1.8936967660000001 podStartE2EDuration="1.893696766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:04.893028308 +0000 UTC m=+649.087551432" watchObservedRunningTime="2026-10-31 03:27:04.893696766 +0000 UTC m=+649.088219907" Oct 31 03:27:05.131943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:05.131855 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:09.020904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:09.020858 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:27:09.033730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:09.033706 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:27:09.366442 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:27:09.366360 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:27:09.931819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:09.931780 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:27:14.132024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:14.131987 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:14.147086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:14.147052 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:14.955034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:14.955003 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:27:33.440103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.440071 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n"] Oct 31 03:27:33.443841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.443825 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.450311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.450289 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-090ebd-openshift-default-dockercfg-kj8g4\"" Oct 31 03:27:33.456697 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.456670 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n"] Oct 31 03:27:33.535742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.535705 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.535915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.535751 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzkg\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-kube-api-access-dxzkg\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.535915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.535858 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.535915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.535903 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.536051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.535920 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.536051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.535982 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.536051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.536027 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.536153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.536083 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.536153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.536118 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.636874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.636841 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.636884 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzkg\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-kube-api-access-dxzkg\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637009 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637076 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637102 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637144 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637181 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637248 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637317 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637473 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637512 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637562 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637701 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.637950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.637930 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.640522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.640503 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.640646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.640625 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.652117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.652089 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.652417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.652395 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzkg\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-kube-api-access-dxzkg\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:33.757469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:33.757374 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:34.044867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:34.044800 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n"] Oct 31 03:27:34.047308 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:27:34.047280 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d842f7b_3ed4_402d_9c2a_2b7a2939c339.slice/crio-8a4e1e24d4656cd7fe275050f556d3793aa8ad98c5e7ca23b2fccdbb214aed2b WatchSource:0}: Error finding container 8a4e1e24d4656cd7fe275050f556d3793aa8ad98c5e7ca23b2fccdbb214aed2b: Status 404 returned error can't find the container with id 8a4e1e24d4656cd7fe275050f556d3793aa8ad98c5e7ca23b2fccdbb214aed2b Oct 31 03:27:34.049694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:34.049662 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:27:34.049798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:34.049740 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:27:34.049798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:34.049786 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:27:34.074002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:34.073965 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" event={"ID":"1d842f7b-3ed4-402d-9c2a-2b7a2939c339","Type":"ContainerStarted","Data":"8a4e1e24d4656cd7fe275050f556d3793aa8ad98c5e7ca23b2fccdbb214aed2b"} Oct 31 03:27:35.098553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:35.098511 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" event={"ID":"1d842f7b-3ed4-402d-9c2a-2b7a2939c339","Type":"ContainerStarted","Data":"a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6"} Oct 31 03:27:35.124129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:35.124073 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podStartSLOduration=2.12405942 podStartE2EDuration="2.12405942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:35.121784814 +0000 UTC m=+679.316307934" watchObservedRunningTime="2026-10-31 03:27:35.12405942 +0000 UTC m=+679.318582540" Oct 31 03:27:35.757679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:35.757643 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:35.759461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:35.759433 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:27:35.759613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:35.759487 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 03:27:36.758225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:36.758174 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:27:36.758616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:36.758265 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 03:27:37.758237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:37.758187 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:27:37.758680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:37.758261 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 03:27:38.758092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:38.758056 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:27:38.758303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:38.758107 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 03:27:39.363235 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:27:39.362043 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:27:39.758169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:39.758138 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 03:27:39.758439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:39.758193 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 03:27:40.763910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:40.763882 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:40.764398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:40.764382 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:27:40.765188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:27:40.765168 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:28:04.304356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.304315 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx"] Oct 31 03:28:04.308042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.308017 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.311486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.311447 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-bcc7a4-openshift-default-dockercfg-pmpv4\"" Oct 31 03:28:04.319936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.319907 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx"] Oct 31 03:28:04.350114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350073 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350149 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1306dc-5094-4f7d-ad20-667064dd914c-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350178 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350224 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350252 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350327 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350614 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350363 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350614 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350424 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.350614 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.350464 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hgz\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-kube-api-access-f7hgz\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451192 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451293 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hgz\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-kube-api-access-f7hgz\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451354 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451423 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1306dc-5094-4f7d-ad20-667064dd914c-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451453 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451483 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451510 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451759 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451839 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.451952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.451933 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.452020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.452004 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.452185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.452164 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.452331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.452315 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.452396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.452377 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1306dc-5094-4f7d-ad20-667064dd914c-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.453736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.453707 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.454047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.454028 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.459263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.459241 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.459446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.459429 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hgz\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-kube-api-access-f7hgz\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.624336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.624240 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:04.906755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.906646 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx"] Oct 31 03:28:04.909898 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:28:04.909869 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1306dc_5094_4f7d_ad20_667064dd914c.slice/crio-e43fcf33ff648c8eb111d31ed98ab80ddd60cfee74ed8888a43ca3c899b6c449 WatchSource:0}: Error finding container e43fcf33ff648c8eb111d31ed98ab80ddd60cfee74ed8888a43ca3c899b6c449: Status 404 returned error can't find the container with id e43fcf33ff648c8eb111d31ed98ab80ddd60cfee74ed8888a43ca3c899b6c449 Oct 31 03:28:04.912969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.912926 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:04.913086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.913015 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:04.913086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:04.913060 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:05.316185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:05.316145 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" event={"ID":"5c1306dc-5094-4f7d-ad20-667064dd914c","Type":"ContainerStarted","Data":"5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75"} Oct 31 03:28:05.316717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:05.316243 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" event={"ID":"5c1306dc-5094-4f7d-ad20-667064dd914c","Type":"ContainerStarted","Data":"e43fcf33ff648c8eb111d31ed98ab80ddd60cfee74ed8888a43ca3c899b6c449"} Oct 31 03:28:05.335803 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:05.335732 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podStartSLOduration=1.335713164 podStartE2EDuration="1.335713164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:05.335348397 +0000 UTC m=+709.529871521" watchObservedRunningTime="2026-10-31 03:28:05.335713164 +0000 UTC m=+709.530236286" Oct 31 03:28:05.629322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:05.628272 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:05.629322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:05.629139 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:28:05.629322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:05.629241 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 03:28:06.625618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:06.625578 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:28:06.626071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:06.625645 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 03:28:07.624921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:07.624877 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:28:07.625131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:07.624939 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 03:28:08.628225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:08.627611 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:28:08.628225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:08.627690 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 03:28:09.362760 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:28:09.362725 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:28:09.624755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:09.624656 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:28:09.625038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:09.625014 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 03:28:10.574363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:10.574310 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:10.627666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:10.627628 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:28:10.627943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:10.627914 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 03:28:10.791443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:10.790421 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:10.796141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:10.795305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:10.910390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:10.910286 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:10.944623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:10.944578 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:11.628235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:11.627586 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 31 03:28:11.628235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:11.627649 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 31 03:28:12.628991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:12.628955 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:12.629430 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:12.629265 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:12.630023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:12.630006 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 03:28:23.806914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.806879 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk"] Oct 31 03:28:23.814477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.814450 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:23.820401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.820361 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk"] Oct 31 03:28:23.820648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.820619 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"distinct-w2-f20dc4-kserve-self-signed-certs\"" Oct 31 03:28:23.964734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.964690 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e483869d-f714-4177-a566-4e6354c11fcf-tls-certs\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:23.964972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.964768 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-model-cache\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:23.964972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.964858 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-dshm\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:23.964972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.964923 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl72g\" (UniqueName: \"kubernetes.io/projected/e483869d-f714-4177-a566-4e6354c11fcf-kube-api-access-wl72g\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:23.965187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.965062 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-home\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:23.965187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:23.965173 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-tmp-dir\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.068681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.067800 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl72g\" (UniqueName: \"kubernetes.io/projected/e483869d-f714-4177-a566-4e6354c11fcf-kube-api-access-wl72g\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.068681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.067896 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-home\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.068681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.067947 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-tmp-dir\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.068681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.068032 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e483869d-f714-4177-a566-4e6354c11fcf-tls-certs\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.068681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.068075 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-model-cache\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.068681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.068114 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-dshm\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.069531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.069499 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-tmp-dir\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.070033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.069997 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-model-cache\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.070356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.070330 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-home\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.082242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.081846 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e483869d-f714-4177-a566-4e6354c11fcf-tls-certs\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.089266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.086970 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-dshm\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.089266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.087597 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl72g\" (UniqueName: \"kubernetes.io/projected/e483869d-f714-4177-a566-4e6354c11fcf-kube-api-access-wl72g\") pod \"distinct-w2-f20dc4-kserve-6fbf86468-lhmjk\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.135866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.135834 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:24.444994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.444965 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk"] Oct 31 03:28:24.448063 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:28:24.448020 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode483869d_f714_4177_a566_4e6354c11fcf.slice/crio-56a61d4a992c2dd7e6bb7b75bbebf4aa3cf0b55a1abf51e6773d2c527a918b33 WatchSource:0}: Error finding container 56a61d4a992c2dd7e6bb7b75bbebf4aa3cf0b55a1abf51e6773d2c527a918b33: Status 404 returned error can't find the container with id 56a61d4a992c2dd7e6bb7b75bbebf4aa3cf0b55a1abf51e6773d2c527a918b33 Oct 31 03:28:24.495750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:24.495699 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" event={"ID":"e483869d-f714-4177-a566-4e6354c11fcf","Type":"ContainerStarted","Data":"56a61d4a992c2dd7e6bb7b75bbebf4aa3cf0b55a1abf51e6773d2c527a918b33"} Oct 31 03:28:25.503332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:25.503240 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" event={"ID":"e483869d-f714-4177-a566-4e6354c11fcf","Type":"ContainerStarted","Data":"dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216"} Oct 31 03:28:25.522708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:25.522657 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podStartSLOduration=2.522643675 podStartE2EDuration="2.522643675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:25.520247501 +0000 UTC m=+729.714770658" watchObservedRunningTime="2026-10-31 03:28:25.522643675 +0000 UTC m=+729.717166796" Oct 31 03:28:26.135994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:26.135950 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:34.136151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.136108 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:34.150356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.150329 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:34.174918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.174881 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk"] Oct 31 03:28:34.179400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.179360 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.192836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.192798 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6c6050-openshift-default-dockercfg-wsmcd\"" Oct 31 03:28:34.196438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.196408 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk"] Oct 31 03:28:34.311534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.311497 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.311748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.311557 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.311805 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.311776 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.311854 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.311838 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16930518-1a71-4a09-9bd0-b139791c0169-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.312044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.312023 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.312133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.312092 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16930518-1a71-4a09-9bd0-b139791c0169-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.312194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.312134 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6c7s\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-kube-api-access-q6c7s\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.312194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.312164 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.312325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.312275 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.413562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413463 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16930518-1a71-4a09-9bd0-b139791c0169-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.413562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413506 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6c7s\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-kube-api-access-q6c7s\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.413562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413529 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.413562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413567 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.413909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413586 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.413909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413614 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.413909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413829 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.414070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.413988 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.414070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.414055 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.414175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.414151 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16930518-1a71-4a09-9bd0-b139791c0169-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.414256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.414233 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.414316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.414249 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.414371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.414316 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16930518-1a71-4a09-9bd0-b139791c0169-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.414634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.414613 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.416412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.416384 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.416554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.416526 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16930518-1a71-4a09-9bd0-b139791c0169-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.421297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.421265 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.421445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.421429 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6c7s\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-kube-api-access-q6c7s\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.494843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.494798 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:34.600470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.598954 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:28:34.781123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.781087 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk"] Oct 31 03:28:34.782592 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:28:34.782559 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16930518_1a71_4a09_9bd0_b139791c0169.slice/crio-6a05cda470f7589a7d4d786f4bc7950ce0f78e04db63e8547f3c534da879309d WatchSource:0}: Error finding container 6a05cda470f7589a7d4d786f4bc7950ce0f78e04db63e8547f3c534da879309d: Status 404 returned error can't find the container with id 6a05cda470f7589a7d4d786f4bc7950ce0f78e04db63e8547f3c534da879309d Oct 31 03:28:34.785460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.785424 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:34.785568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.785506 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:34.785568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:34.785549 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:35.612877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:35.612268 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" event={"ID":"16930518-1a71-4a09-9bd0-b139791c0169","Type":"ContainerStarted","Data":"89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c"} Oct 31 03:28:35.612877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:35.612360 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" event={"ID":"16930518-1a71-4a09-9bd0-b139791c0169","Type":"ContainerStarted","Data":"6a05cda470f7589a7d4d786f4bc7950ce0f78e04db63e8547f3c534da879309d"} Oct 31 03:28:35.631603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:35.631547 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podStartSLOduration=1.631534797 podStartE2EDuration="1.631534797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:35.630995896 +0000 UTC m=+739.825519046" watchObservedRunningTime="2026-10-31 03:28:35.631534797 +0000 UTC m=+739.826057918" Oct 31 03:28:36.497226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:36.496003 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:36.505225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:36.501312 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:28:36.505225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:36.501383 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 03:28:37.492230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.491547 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz"] Oct 31 03:28:37.503224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.500672 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:28:37.503224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.500726 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 03:28:37.512178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.508917 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.512178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.510315 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz"] Oct 31 03:28:37.522458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.522435 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"distinct-2-w6-7f3159-kserve-self-signed-certs\"" Oct 31 03:28:37.666999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.666964 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-home\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.667372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.667339 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knq7q\" (UniqueName: \"kubernetes.io/projected/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-kube-api-access-knq7q\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.667606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.667590 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-dshm\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.667757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.667741 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-model-cache\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.667997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.667980 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tls-certs\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.668301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.668276 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tmp-dir\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.772112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.772013 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-home\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.772480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.772457 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knq7q\" (UniqueName: \"kubernetes.io/projected/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-kube-api-access-knq7q\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.772615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.772599 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-dshm\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.772779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.772765 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-model-cache\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.772894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.772882 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tls-certs\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.773004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.772993 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tmp-dir\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.774088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.774060 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-home\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.781322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.774857 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tmp-dir\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.781523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.775591 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-model-cache\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.798333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.782404 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tls-certs\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.798558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.789545 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-dshm\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.798690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.790266 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knq7q\" (UniqueName: \"kubernetes.io/projected/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-kube-api-access-knq7q\") pod \"distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:37.835492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:37.835367 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:38.198322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:38.198157 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz"] Oct 31 03:28:38.202971 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:28:38.202937 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod151ddd1e_4ecc_4bb7_80f6_e0b630375cab.slice/crio-3e858075b5d54f0ac605b37cccfcdb800f21795fcfebc68db629c33c993d2386 WatchSource:0}: Error finding container 3e858075b5d54f0ac605b37cccfcdb800f21795fcfebc68db629c33c993d2386: Status 404 returned error can't find the container with id 3e858075b5d54f0ac605b37cccfcdb800f21795fcfebc68db629c33c993d2386 Oct 31 03:28:38.495713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:38.495609 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:28:38.495713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:38.495674 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 03:28:38.637463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:38.637425 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" event={"ID":"151ddd1e-4ecc-4bb7-80f6-e0b630375cab","Type":"ContainerStarted","Data":"40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676"} Oct 31 03:28:38.637664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:38.637474 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" event={"ID":"151ddd1e-4ecc-4bb7-80f6-e0b630375cab","Type":"ContainerStarted","Data":"3e858075b5d54f0ac605b37cccfcdb800f21795fcfebc68db629c33c993d2386"} Oct 31 03:28:38.658325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:38.658279 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podStartSLOduration=1.658265938 podStartE2EDuration="1.658265938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:38.656094615 +0000 UTC m=+742.850617766" watchObservedRunningTime="2026-10-31 03:28:38.658265938 +0000 UTC m=+742.852789058" Oct 31 03:28:38.840461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:38.839354 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:39.359748 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:28:39.359704 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:28:39.499225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:39.498623 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:28:39.499225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:39.498700 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 03:28:40.498231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:40.495631 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:28:40.498231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:40.495703 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 03:28:41.496000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:41.495958 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:28:41.496483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:41.496043 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 03:28:42.500379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:42.500342 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:42.500993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:42.500764 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:42.501437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:42.501419 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:28:44.382550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.382515 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz"] Oct 31 03:28:44.386693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.386662 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.396508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.396482 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"distinct-2-w2-f20dc4-kserve-self-signed-certs\"" Oct 31 03:28:44.398108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.398077 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz"] Oct 31 03:28:44.499377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.499332 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-home\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.499597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.499410 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-dshm\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.499597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.499520 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tls-certs\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.499725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.499605 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tmp-dir\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.499725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.499662 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmmvm\" (UniqueName: \"kubernetes.io/projected/af776a7f-f7b5-4392-a420-dcc54e7dd0df-kube-api-access-lmmvm\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.499725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.499721 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-model-cache\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.605880 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-home\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.605953 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-dshm\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.605992 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tls-certs\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.606053 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tmp-dir\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.606105 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmvm\" (UniqueName: \"kubernetes.io/projected/af776a7f-f7b5-4392-a420-dcc54e7dd0df-kube-api-access-lmmvm\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.606162 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-model-cache\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.606634 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-model-cache\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.606856 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-home\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.609230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.607488 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tmp-dir\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.618337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.618307 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-dshm\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.620830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.620802 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmvm\" (UniqueName: \"kubernetes.io/projected/af776a7f-f7b5-4392-a420-dcc54e7dd0df-kube-api-access-lmmvm\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.645682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.645252 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tls-certs\") pod \"distinct-2-w2-f20dc4-kserve-858d745599-765tz\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.705416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.705381 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:44.993355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:44.993327 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz"] Oct 31 03:28:44.995281 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:28:44.995249 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf776a7f_f7b5_4392_a420_dcc54e7dd0df.slice/crio-27a0eb4de4b01785b3c921c1d11654d447250254b546410b71fcb12b1e1f0fd4 WatchSource:0}: Error finding container 27a0eb4de4b01785b3c921c1d11654d447250254b546410b71fcb12b1e1f0fd4: Status 404 returned error can't find the container with id 27a0eb4de4b01785b3c921c1d11654d447250254b546410b71fcb12b1e1f0fd4 Oct 31 03:28:45.686579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:45.686535 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" event={"ID":"af776a7f-f7b5-4392-a420-dcc54e7dd0df","Type":"ContainerStarted","Data":"96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3"} Oct 31 03:28:45.687043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:45.686589 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" event={"ID":"af776a7f-f7b5-4392-a420-dcc54e7dd0df","Type":"ContainerStarted","Data":"27a0eb4de4b01785b3c921c1d11654d447250254b546410b71fcb12b1e1f0fd4"} Oct 31 03:28:45.704405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:45.704358 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podStartSLOduration=1.704345059 podStartE2EDuration="1.704345059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:45.703041171 +0000 UTC m=+749.897564314" watchObservedRunningTime="2026-10-31 03:28:45.704345059 +0000 UTC m=+749.898868180" Oct 31 03:28:45.706376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:45.706354 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:47.835554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:47.835513 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:47.848809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:47.848785 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:48.717031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:48.716999 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:28:49.607243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.607190 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv"] Oct 31 03:28:49.611189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.611169 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.617151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.617120 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-dda5cb-openshift-default-dockercfg-z7cvq\"" Oct 31 03:28:49.623183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.623158 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv"] Oct 31 03:28:49.683839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.683800 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.683850 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.683963 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.684011 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d903c2-8f71-46a3-8523-930b333f870c-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.684106 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d903c2-8f71-46a3-8523-930b333f870c-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.684168 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.684245 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.684293 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.684448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.684421 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8n2\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-kube-api-access-zj8n2\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785155 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8n2\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-kube-api-access-zj8n2\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785254 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785282 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785332 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785380 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d903c2-8f71-46a3-8523-930b333f870c-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785442 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d903c2-8f71-46a3-8523-930b333f870c-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785470 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785499 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785534 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.785815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785717 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.786150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.785979 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.786150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.786070 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.786150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.786109 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.786554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.786349 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d903c2-8f71-46a3-8523-930b333f870c-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.788263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.788241 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.788700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.788679 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d903c2-8f71-46a3-8523-930b333f870c-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.804614 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.804586 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8n2\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-kube-api-access-zj8n2\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.805898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.805866 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:49.931226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:49.930396 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:50.285502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.282846 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv"] Oct 31 03:28:50.292466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.287101 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:50.292466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.287270 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:50.292466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.287354 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:28:50.744228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.734949 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" event={"ID":"19d903c2-8f71-46a3-8523-930b333f870c","Type":"ContainerStarted","Data":"69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d"} Oct 31 03:28:50.744228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.735022 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" event={"ID":"19d903c2-8f71-46a3-8523-930b333f870c","Type":"ContainerStarted","Data":"f48c5228f913f4346a3e945b4fefce35bdd7a35662a3e2e944aca6d4bf3ab590"} Oct 31 03:28:50.788410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.788337 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podStartSLOduration=1.788315648 podStartE2EDuration="1.788315648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:50.779355402 +0000 UTC m=+754.973878525" watchObservedRunningTime="2026-10-31 03:28:50.788315648 +0000 UTC m=+754.982838770" Oct 31 03:28:50.938339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:50.938307 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:51.932944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:51.932907 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:28:51.933369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:51.932987 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 03:28:52.933227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:52.932596 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:28:52.933227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:52.932655 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 03:28:53.932635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:53.932594 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:28:53.932774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:53.932660 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 03:28:54.711810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:54.711780 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:54.763326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:54.763294 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:54.819106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:54.819075 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:28:54.931969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:54.931936 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:28:54.932152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:54.931993 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 03:28:55.931395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:55.931355 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:28:55.931806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:55.931419 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 03:28:56.943227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:56.934562 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 03:28:56.943227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:56.934634 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 03:28:58.120368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:58.120336 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:58.120891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:58.120744 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:28:58.121751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:28:58.121732 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:29:03.857325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.857286 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc"] Oct 31 03:29:03.870312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.867378 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.875515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.875488 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f20dc4-openshift-default-dockercfg-zn4w2\"" Oct 31 03:29:03.876869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.876810 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc"] Oct 31 03:29:03.932520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932485 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932528 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49484854-701e-49f5-9aa0-85e2338de33c-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932619 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932656 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932744 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt49k\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-kube-api-access-bt49k\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932795 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932831 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932849 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:03.932915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:03.932869 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49484854-701e-49f5-9aa0-85e2338de33c-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034507 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034566 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49484854-701e-49f5-9aa0-85e2338de33c-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034615 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034665 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034725 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt49k\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-kube-api-access-bt49k\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034766 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034817 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034845 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.034947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034877 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49484854-701e-49f5-9aa0-85e2338de33c-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.035151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.034977 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.035151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.035073 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.035309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.035289 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.035387 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.035365 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.035506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.035478 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49484854-701e-49f5-9aa0-85e2338de33c-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.037426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.037403 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.037665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.037646 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49484854-701e-49f5-9aa0-85e2338de33c-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.043171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.043140 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.043753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.043730 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt49k\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-kube-api-access-bt49k\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.194326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.194287 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:04.480678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.480644 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc"] Oct 31 03:29:04.494240 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:29:04.482511 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49484854_701e_49f5_9aa0_85e2338de33c.slice/crio-e3c3c00c9dca9c54825ecee9c4c21bbd51826b37b83603d19e6f7312f323e368 WatchSource:0}: Error finding container e3c3c00c9dca9c54825ecee9c4c21bbd51826b37b83603d19e6f7312f323e368: Status 404 returned error can't find the container with id e3c3c00c9dca9c54825ecee9c4c21bbd51826b37b83603d19e6f7312f323e368 Oct 31 03:29:04.513865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.498084 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:04.513865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.498184 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:04.515077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.498247 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:04.916911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.916811 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" event={"ID":"49484854-701e-49f5-9aa0-85e2338de33c","Type":"ContainerStarted","Data":"f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104"} Oct 31 03:29:04.916911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.916875 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" event={"ID":"49484854-701e-49f5-9aa0-85e2338de33c","Type":"ContainerStarted","Data":"e3c3c00c9dca9c54825ecee9c4c21bbd51826b37b83603d19e6f7312f323e368"} Oct 31 03:29:04.941429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:04.940787 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podStartSLOduration=1.940766515 podStartE2EDuration="1.940766515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:04.937566702 +0000 UTC m=+769.132089824" watchObservedRunningTime="2026-10-31 03:29:04.940766515 +0000 UTC m=+769.135289639" Oct 31 03:29:05.200228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:05.198905 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:05.202380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:05.202347 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:29:05.202613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:05.202590 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 03:29:06.198227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:06.197614 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:29:06.198227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:06.197678 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 03:29:07.198225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:07.197644 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:29:07.198225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:07.197711 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 03:29:08.199170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:08.195658 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:29:08.199170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:08.195725 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 03:29:09.194944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:09.194903 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:29:09.195298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:09.195265 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 03:29:09.360574 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:29:09.360536 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:29:10.195624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:10.195587 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:29:10.196101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:10.196062 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 03:29:11.198725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:11.198679 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 31 03:29:11.199376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:11.199340 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 31 03:29:12.586500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:12.586433 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:12.756393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:12.756340 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:12.912694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:12.912577 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:29:13.213277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:13.210466 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:29:13.213277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:13.210541 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:13.459888 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:13.459848 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:13.460830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:13.460811 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:13.462928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:13.462912 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:29:14.007296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.007258 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx"] Oct 31 03:29:14.023671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.023643 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.030700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.030658 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx"] Oct 31 03:29:14.033772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.033736 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"distinct-w4-dda5cb-kserve-self-signed-certs\"" Oct 31 03:29:14.136227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.135588 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dcw\" (UniqueName: \"kubernetes.io/projected/63146752-5307-488c-9217-8d41a8e28e29-kube-api-access-d8dcw\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.136227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.135640 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63146752-5307-488c-9217-8d41a8e28e29-tls-certs\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.136227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.135710 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-model-cache\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.136227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.135796 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-dshm\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.136227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.135849 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-tmp-dir\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.136227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.135918 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-home\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.239083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.239040 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-model-cache\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.239978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.239957 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-dshm\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.241018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.240996 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-tmp-dir\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.241529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.239785 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-model-cache\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.241658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.241427 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-tmp-dir\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.241978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.241958 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-home\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.242183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.242156 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dcw\" (UniqueName: \"kubernetes.io/projected/63146752-5307-488c-9217-8d41a8e28e29-kube-api-access-d8dcw\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.242391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.242362 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63146752-5307-488c-9217-8d41a8e28e29-tls-certs\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.243466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.243444 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-home\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.262900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.262607 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63146752-5307-488c-9217-8d41a8e28e29-tls-certs\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.264310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.263656 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dcw\" (UniqueName: \"kubernetes.io/projected/63146752-5307-488c-9217-8d41a8e28e29-kube-api-access-d8dcw\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.294452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.294421 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-dshm\") pod \"distinct-w4-dda5cb-kserve-cdb6b8946-98jjx\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.348119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.348081 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:14.681374 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:29:14.681342 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63146752_5307_488c_9217_8d41a8e28e29.slice/crio-b1c00053ce42a38baaedee2cc5e2ba2e510f7778fdeffaded848af08ceda24f2 WatchSource:0}: Error finding container b1c00053ce42a38baaedee2cc5e2ba2e510f7778fdeffaded848af08ceda24f2: Status 404 returned error can't find the container with id b1c00053ce42a38baaedee2cc5e2ba2e510f7778fdeffaded848af08ceda24f2 Oct 31 03:29:14.726254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:14.705571 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx"] Oct 31 03:29:15.155174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:15.155131 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" event={"ID":"63146752-5307-488c-9217-8d41a8e28e29","Type":"ContainerStarted","Data":"3251d6567de7a077d7fb08ec4bc0dc0f785a1b12fb8eec665233896ff068988b"} Oct 31 03:29:15.155174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:15.155178 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" event={"ID":"63146752-5307-488c-9217-8d41a8e28e29","Type":"ContainerStarted","Data":"b1c00053ce42a38baaedee2cc5e2ba2e510f7778fdeffaded848af08ceda24f2"} Oct 31 03:29:15.173040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:15.172985 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podStartSLOduration=2.172969693 podStartE2EDuration="2.172969693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:15.170811603 +0000 UTC m=+779.365334724" watchObservedRunningTime="2026-10-31 03:29:15.172969693 +0000 UTC m=+779.367492814" Oct 31 03:29:15.348572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:15.348539 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:18.672058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.672024 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7"] Oct 31 03:29:18.677811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.677783 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.701221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.701172 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7"] Oct 31 03:29:18.710169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710139 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710189 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710244 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710396 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710443 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710529 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2jl2\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-kube-api-access-b2jl2\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710581 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710625 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.710687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.710674 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811685 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811723 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811774 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jl2\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-kube-api-access-b2jl2\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811803 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811828 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811868 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811938 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.811969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811968 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.812318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.811996 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.812318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.812303 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.812425 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.812404 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.812734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.812667 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.812734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.812720 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.812734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.812737 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.814493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.814471 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.814700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.814677 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.820906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.820885 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:18.821170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:18.821135 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jl2\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-kube-api-access-b2jl2\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:19.003093 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.003054 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:19.095449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.094816 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx"] Oct 31 03:29:19.101822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.101795 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.109290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.109258 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d86a77-openshift-default-dockercfg-2xd8c\"" Oct 31 03:29:19.156275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.144417 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx"] Oct 31 03:29:19.217397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217355 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217518 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217566 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217663 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217692 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217774 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217803 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcz4\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-kube-api-access-qwcz4\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217874 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.217963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.217899 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.296671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.295558 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7"] Oct 31 03:29:19.299916 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:29:19.298916 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ec6ff4_0c20_48bd_bce8_4e7377bc0073.slice/crio-32f4ff9c8bc84bfd9ad2f646da31cc3a169b8304e091b912b004f0eae1b87820 WatchSource:0}: Error finding container 32f4ff9c8bc84bfd9ad2f646da31cc3a169b8304e091b912b004f0eae1b87820: Status 404 returned error can't find the container with id 32f4ff9c8bc84bfd9ad2f646da31cc3a169b8304e091b912b004f0eae1b87820 Oct 31 03:29:19.302183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.302092 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:19.302322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.302181 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:19.302322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.302249 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:19.319235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319176 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319232 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcz4\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-kube-api-access-qwcz4\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319284 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319300 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319343 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319371 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319390 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319459 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.319481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.319477 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.320118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.320089 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.320299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.320280 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.320559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.320499 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.320656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.320572 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.320988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.320969 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.322286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.322264 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.322842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.322823 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.327886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.327864 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.328118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.328100 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcz4\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-kube-api-access-qwcz4\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.417619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.417588 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:19.445252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.440905 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh"] Oct 31 03:29:19.449227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.445815 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.473772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.471763 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh"] Oct 31 03:29:19.523559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523446 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.523559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523517 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.523801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523585 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.523801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523677 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.523801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523770 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.524056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523817 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.524056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523863 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pn5v\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-kube-api-access-8pn5v\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.524056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523936 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.524056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.523991 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.625932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.625837 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.625932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.625882 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.625932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.625927 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.625968 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pn5v\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-kube-api-access-8pn5v\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626027 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626074 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626124 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626173 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626542 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626282 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626542 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626349 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626750 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626756 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.626895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.626817 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.627411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.627386 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.629622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.629582 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.629785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.629760 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.634523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.634499 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pn5v\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-kube-api-access-8pn5v\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.634743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.634726 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.741765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.741681 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx"] Oct 31 03:29:19.747393 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:29:19.747355 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d58c9a_ac48_41b7_bf68_c54fbea4531e.slice/crio-076557dafa987a70aff1d969aad1a0b1390f8be8ab9e5051dbee36f627810b1e WatchSource:0}: Error finding container 076557dafa987a70aff1d969aad1a0b1390f8be8ab9e5051dbee36f627810b1e: Status 404 returned error can't find the container with id 076557dafa987a70aff1d969aad1a0b1390f8be8ab9e5051dbee36f627810b1e Oct 31 03:29:19.750609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.750575 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:19.750729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.750654 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:19.750729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.750685 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:19.779761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.779729 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f"] Oct 31 03:29:19.781047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.781016 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:19.789784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.786362 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.813224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.810305 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f"] Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.830933 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.830992 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.831024 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.831082 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.831259 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.831292 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e460fac-4fc9-47ca-af24-19f08f093c46-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.831372 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.831421 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vh4\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-kube-api-access-p9vh4\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.853874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.831452 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.933875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.933848 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934288 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vh4\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-kube-api-access-p9vh4\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934331 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934387 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934420 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934453 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934514 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934665 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.934697 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e460fac-4fc9-47ca-af24-19f08f093c46-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.935581 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e460fac-4fc9-47ca-af24-19f08f093c46-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.935893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.935872 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.937302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.936728 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.937302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.937038 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.937302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.937185 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.944483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.944427 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.945643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.945551 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.946182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.946155 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vh4\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-kube-api-access-p9vh4\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:19.947620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:19.947597 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:20.123485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.123186 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:20.208127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.204613 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh"] Oct 31 03:29:20.213263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.212737 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:20.213263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.212820 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:20.213263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.212878 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:20.221380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.220678 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" event={"ID":"a8d58c9a-ac48-41b7-bf68-c54fbea4531e","Type":"ContainerStarted","Data":"9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541"} Oct 31 03:29:20.221380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.220728 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" event={"ID":"a8d58c9a-ac48-41b7-bf68-c54fbea4531e","Type":"ContainerStarted","Data":"076557dafa987a70aff1d969aad1a0b1390f8be8ab9e5051dbee36f627810b1e"} Oct 31 03:29:20.223554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.222941 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" event={"ID":"09ec6ff4-0c20-48bd-bce8-4e7377bc0073","Type":"ContainerStarted","Data":"b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24"} Oct 31 03:29:20.223554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.222989 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" event={"ID":"09ec6ff4-0c20-48bd-bce8-4e7377bc0073","Type":"ContainerStarted","Data":"32f4ff9c8bc84bfd9ad2f646da31cc3a169b8304e091b912b004f0eae1b87820"} Oct 31 03:29:20.250914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.249303 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podStartSLOduration=1.249287898 podStartE2EDuration="1.249287898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:20.245330292 +0000 UTC m=+784.439853415" watchObservedRunningTime="2026-10-31 03:29:20.249287898 +0000 UTC m=+784.443811019" Oct 31 03:29:20.419229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.419140 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:20.420846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.420459 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:20.420846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.420519 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:20.523036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.521037 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podStartSLOduration=2.5210168939999997 podStartE2EDuration="2.521016894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:20.320725628 +0000 UTC m=+784.515248776" watchObservedRunningTime="2026-10-31 03:29:20.521016894 +0000 UTC m=+784.715540017" Oct 31 03:29:20.523036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.522506 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f"] Oct 31 03:29:20.529732 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:29:20.529053 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e460fac_4fc9_47ca_af24_19f08f093c46.slice/crio-d359ec4371524439f0da8b6e0a6e80b594ef587db75e06d3d44877696f8c772e WatchSource:0}: Error finding container d359ec4371524439f0da8b6e0a6e80b594ef587db75e06d3d44877696f8c772e: Status 404 returned error can't find the container with id d359ec4371524439f0da8b6e0a6e80b594ef587db75e06d3d44877696f8c772e Oct 31 03:29:20.532340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.532251 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:20.532574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.532552 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:20.532727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:20.532710 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:21.003370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.003340 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:21.007490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.007258 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:21.007825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.007794 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:21.245963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.245926 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" event={"ID":"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd","Type":"ContainerStarted","Data":"ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973"} Oct 31 03:29:21.246294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.246268 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" event={"ID":"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd","Type":"ContainerStarted","Data":"30e7876e3f1f9d533214c4a495ff2e8b332a9df1c4cadeb9bbbcd148b12f19ba"} Oct 31 03:29:21.253232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.251520 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" event={"ID":"2e460fac-4fc9-47ca-af24-19f08f093c46","Type":"ContainerStarted","Data":"61a8302d544c25e4fc2518d5eb8ff6761c681c1b60d5af54084394ed9d575a8f"} Oct 31 03:29:21.253232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.251571 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" event={"ID":"2e460fac-4fc9-47ca-af24-19f08f093c46","Type":"ContainerStarted","Data":"d359ec4371524439f0da8b6e0a6e80b594ef587db75e06d3d44877696f8c772e"} Oct 31 03:29:21.301691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.301574 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podStartSLOduration=2.301556573 podStartE2EDuration="2.301556573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:21.299747796 +0000 UTC m=+785.494270933" watchObservedRunningTime="2026-10-31 03:29:21.301556573 +0000 UTC m=+785.496079694" Oct 31 03:29:21.317287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.306045 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podStartSLOduration=2.306030254 podStartE2EDuration="2.306030254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:21.28008517 +0000 UTC m=+785.474608288" watchObservedRunningTime="2026-10-31 03:29:21.306030254 +0000 UTC m=+785.500553367" Oct 31 03:29:21.421491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.421458 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:21.421768 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.421745 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:21.783233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.782530 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:21.791226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.788313 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:21.791226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.788374 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:21.873230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:21.872299 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:22.004111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.003658 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:22.004111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.003723 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:22.139001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.138905 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:22.171972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.171916 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:22.172286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.172261 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:22.419905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.419812 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:22.420143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.420116 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:22.710407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.710348 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:22.787860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.787828 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:22.788122 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.788100 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:22.883839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.883801 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:22.951525 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:22.950354 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:23.004247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.003714 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:23.004247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.003777 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:23.131226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.126605 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:23.131226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.126680 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:23.427286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.426655 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:23.427286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.426717 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:23.586694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.585579 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:23.797229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.784347 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:23.797229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.785085 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:23.797229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.786663 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:23.797229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.787896 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:23.804231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.803680 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:29:23.877236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.819351 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:23.892235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.877504 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:23.892235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.819997 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:23.938352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.937002 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:23.970239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:23.964953 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:29:24.023340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.023303 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:24.055460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.035034 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:24.064235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.055274 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:24.248267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.243689 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:24.248267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.243748 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:24.355283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.344018 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 31 03:29:24.380228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.358166 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:24.429156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.429121 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:24.429434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.429404 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:24.585376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.585325 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:29:24.640930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.640358 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:24.784670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.784621 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:24.786016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.785989 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:24.786258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.786192 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:24.790306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.790278 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:29:24.790782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.790764 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:29:24.817945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.817905 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:24.818832 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.818808 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:29:24.997344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:24.997303 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:29:25.000520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.000489 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:29:25.004235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.000753 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:29:25.027240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.005560 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:25.027240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.005616 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:25.034270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.034233 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:25.130073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.130038 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:25.130400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.130373 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:25.235632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.235588 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:25.236223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.235879 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:25.312594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.312504 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:25.357336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.357283 5520 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:25.453261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.426355 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:25.454401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.454362 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:25.588586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.588497 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:25.714228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.713299 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:25.783608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.783571 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:25.783851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.783825 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:25.785409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.785381 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:29:25.800627 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.800588 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:25.819319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.819276 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:29:25.998229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:25.997673 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.006280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.005824 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:26.006280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.005880 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:26.127098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.126661 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:26.127098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.126720 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:26.426912 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.426806 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:26.427578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.427535 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:26.470321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.470282 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.576847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.576798 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.591624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.591584 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:29:26.592545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.592521 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:29:26.641309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.641268 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.781419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.781361 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.790249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.782755 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:26.790249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.782805 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:26.819255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.804547 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.819255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.804905 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:29:26.819255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.818713 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.824262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.819618 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.884257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.877305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:26.885242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.884584 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:29:26.986933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:26.986883 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:27.011307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.000546 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:27.025134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.014152 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:27.080479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.080383 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:27.160229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.144166 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:27.160229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.144526 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:27.423284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.420686 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:27.423284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.420745 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:27.600232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.590316 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:27.600232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.591079 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:29:27.725229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.716629 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:27.725229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.717389 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:29:27.757234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.756371 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:27.791276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.784551 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:29:27.860228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.857384 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:27.860228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.858799 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:29:27.889287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.879222 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:29:27.889287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.881102 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:29:27.918252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.891712 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:29:27.918252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.892047 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:29:27.918252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.892362 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:29:27.999312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:27.996821 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:28.012453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.006733 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:28.012453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.006793 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:28.012453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.007124 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:28.012453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.007164 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:28.155358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.045011 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:29:28.416288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.411378 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:29:28.416288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.126846 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:28.416288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.412455 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:28.416288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.150633 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:28.426177 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.151684 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:28.783918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.772491 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:28.783918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.772553 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:28.886980 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.886934 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:28.955247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.949628 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:28.955247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:28.949689 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:29.116104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.115995 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:29.116829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.116801 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:29.154606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.133815 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:29:29.169317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.133853 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:29:29.195228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.169971 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:29:29.213233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.143571 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:29.213233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.143671 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 31 03:29:29.213233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.140329 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:29.213233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.210693 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:29.242228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.241523 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:29.278404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.269056 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:29.278404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.269118 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:29.320251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.312887 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:29:29.414723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.414625 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:29:29.482531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.482498 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:29.483151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.483107 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:29.726228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.725317 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:29.734229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.731355 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:29.785237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.784289 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:29.785237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.784361 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:29.814240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.813927 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:29.818345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.818304 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:29.865233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.864832 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:29:29.875237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.871355 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:29:29.925085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:29.925033 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:30.117296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.111141 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 31 03:29:30.117296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.111226 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 31 03:29:30.157359 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.157307 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:30.166401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.164338 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:30.194835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.194800 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:30.195122 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.195082 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:30.196719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.196689 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:29:30.219043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.218996 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:30.236661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.235557 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:30.236972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.236959 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:30.417285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.389502 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:29:30.442694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.438150 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 03:29:30.446783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.446740 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 03:29:30.562412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.556676 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:29:30.644341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.643174 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:29:30.710129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.710101 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:30.741367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.741337 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:29:30.843367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.825880 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 03:29:30.891310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:30.874066 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 03:29:31.124700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.124608 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 03:29:31.124963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.124933 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 03:29:31.434538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.434474 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:31.435639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.435143 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:31.436588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.436565 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:29:31.564535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.561975 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:31.567085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.566293 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:31.567085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:31.567040 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:29:32.211666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:32.211637 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:32.212501 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:32.212475 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:32.214070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:32.214053 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:29:32.487963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:32.487739 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:32.488808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:32.488786 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:32.489667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:32.489646 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:29:33.703560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.703526 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr"] Oct 31 03:29:33.708551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.708514 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.726873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.726179 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr"] Oct 31 03:29:33.804004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.803968 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjkv\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-kube-api-access-9rjkv\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804107 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804173 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804263 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804291 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804423 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804480 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804545 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.804591 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.804572 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e030301-62ab-4734-9bcf-cd9a0854817c-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906580 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjkv\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-kube-api-access-9rjkv\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906700 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906729 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906778 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906811 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906862 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906907 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.906990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.906953 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.907252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.907003 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e030301-62ab-4734-9bcf-cd9a0854817c-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.908631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.908604 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.908793 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.908770 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.908974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.908949 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.909120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.909084 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.910080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.910057 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e030301-62ab-4734-9bcf-cd9a0854817c-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.918957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.918927 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.920320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.920300 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.921189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.921170 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjkv\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-kube-api-access-9rjkv\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:33.923182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:33.922424 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:34.034935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.034845 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:34.115807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.115739 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj"] Oct 31 03:29:34.130667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.129704 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.135054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.135025 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj"] Oct 31 03:29:34.214513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214483 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.214666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214532 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.214666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214651 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa1d260-f10c-4faf-9a19-119d95aa59a6-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.214771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214708 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6vp\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-kube-api-access-6q6vp\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.214771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214762 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.214846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214822 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.214985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214968 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.215024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.214995 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.215024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.215012 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.272012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.271980 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv"] Oct 31 03:29:34.279127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.279097 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.288174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.288142 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv"] Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.316265 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6vp\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-kube-api-access-6q6vp\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.316313 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.316390 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.316425 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794074d9-ca09-408d-be79-9ebd51d93c81-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.316461 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794074d9-ca09-408d-be79-9ebd51d93c81-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.316565 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.316594 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317113 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317365 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317402 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317442 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317481 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317508 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317530 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317568 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317592 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317635 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317681 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa1d260-f10c-4faf-9a19-119d95aa59a6-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317731 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2798\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-kube-api-access-g2798\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.318291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.317796 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.320645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.320615 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.320924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.320899 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.321109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.321086 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa1d260-f10c-4faf-9a19-119d95aa59a6-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.322450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.322425 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.322727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.322628 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.324595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.324573 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6vp\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-kube-api-access-6q6vp\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.325321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.325303 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.338290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.338269 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr"] Oct 31 03:29:34.343910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.343883 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:34.343986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.343945 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:34.343986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.343972 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:34.363635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.363613 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:34.381504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.381481 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:34.420098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420061 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.420293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420136 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2798\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-kube-api-access-g2798\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.420293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420177 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.420424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420367 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794074d9-ca09-408d-be79-9ebd51d93c81-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.420424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420403 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794074d9-ca09-408d-be79-9ebd51d93c81-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.420547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420528 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.420623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420594 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.421282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420620 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.421282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.420672 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.421282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.421083 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.421802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.421779 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.423216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.423177 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794074d9-ca09-408d-be79-9ebd51d93c81-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.423681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.423660 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.424282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.424261 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.426265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.425358 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794074d9-ca09-408d-be79-9ebd51d93c81-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.426585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.426506 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.429758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.429714 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2798\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-kube-api-access-g2798\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.429947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.429926 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.466930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.466859 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:34.569983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.569948 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" event={"ID":"2e030301-62ab-4734-9bcf-cd9a0854817c","Type":"ContainerStarted","Data":"b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd"} Oct 31 03:29:34.570115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.570007 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" event={"ID":"2e030301-62ab-4734-9bcf-cd9a0854817c","Type":"ContainerStarted","Data":"c12ee05bc9307ae9bea481af64697d654167b7b653748d6cd72b72ca957d7517"} Oct 31 03:29:34.596009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.593120 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podStartSLOduration=1.593101433 podStartE2EDuration="1.593101433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:34.59093567 +0000 UTC m=+798.785458830" watchObservedRunningTime="2026-10-31 03:29:34.593101433 +0000 UTC m=+798.787624555" Oct 31 03:29:34.603784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.603428 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:34.738242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.732307 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p"] Oct 31 03:29:34.747244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.747215 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.841424 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.841604 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.842099 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.842131 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.842377 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.842588 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.843136 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.843178 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnqjz\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-kube-api-access-gnqjz\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.843359 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.855996 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.856058 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:34.856662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.856087 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:34.884556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.884454 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p"] Oct 31 03:29:34.890116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.890000 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj"] Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944371 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944419 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnqjz\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-kube-api-access-gnqjz\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944556 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944646 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944670 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944711 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944732 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944754 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.944781 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.945596 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.947296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.946106 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.948664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.948392 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.948664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.948438 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.949231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.949175 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.949231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.949186 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.953139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.953120 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.959311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.959290 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnqjz\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-kube-api-access-gnqjz\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:34.979297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:34.975642 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:35.035011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.034979 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:35.037000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.036976 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:35.037246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.037184 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:35.144434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.144357 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:35.163284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.163249 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv"] Oct 31 03:29:35.196997 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:29:35.196961 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794074d9_ca09_408d_be79_9ebd51d93c81.slice/crio-409f3a61811f13594243a9f3835cdea17ffca87278a0d0f4bf2b2b81c52b2607 WatchSource:0}: Error finding container 409f3a61811f13594243a9f3835cdea17ffca87278a0d0f4bf2b2b81c52b2607: Status 404 returned error can't find the container with id 409f3a61811f13594243a9f3835cdea17ffca87278a0d0f4bf2b2b81c52b2607 Oct 31 03:29:35.200174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.200136 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:35.200284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.200249 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:35.200349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.200317 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:35.594475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.590911 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:35.594475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.591080 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:35.594475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.591256 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:29:35.643957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.639474 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" event={"ID":"caa1d260-f10c-4faf-9a19-119d95aa59a6","Type":"ContainerStarted","Data":"201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97"} Oct 31 03:29:35.674132 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.674072 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" event={"ID":"caa1d260-f10c-4faf-9a19-119d95aa59a6","Type":"ContainerStarted","Data":"c0093b38b79d9ca1e70d95a83f13d68cb2e0646eadb1ed063f91683b94f2d81f"} Oct 31 03:29:35.676728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.676694 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" event={"ID":"794074d9-ca09-408d-be79-9ebd51d93c81","Type":"ContainerStarted","Data":"bfa17b7e94d07ad543e20d1bbdb5e059da5a46606aff8967c8e383e05b9ba984"} Oct 31 03:29:35.683148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.683122 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p"] Oct 31 03:29:35.683389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.683366 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" event={"ID":"794074d9-ca09-408d-be79-9ebd51d93c81","Type":"ContainerStarted","Data":"409f3a61811f13594243a9f3835cdea17ffca87278a0d0f4bf2b2b81c52b2607"} Oct 31 03:29:35.683544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.683522 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" event={"ID":"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9","Type":"ContainerStarted","Data":"126e34995ffd1d91a7fa179a133108579647d6ba7c51e233cf77e5e77b7875f1"} Oct 31 03:29:35.695730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.695674 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podStartSLOduration=1.695656133 podStartE2EDuration="1.695656133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.695066089 +0000 UTC m=+799.889589227" watchObservedRunningTime="2026-10-31 03:29:35.695656133 +0000 UTC m=+799.890179256" Oct 31 03:29:35.753483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:35.753374 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podStartSLOduration=1.753355504 podStartE2EDuration="1.753355504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.751741046 +0000 UTC m=+799.946264167" watchObservedRunningTime="2026-10-31 03:29:35.753355504 +0000 UTC m=+799.947878625" Oct 31 03:29:36.041283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.037942 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:36.041283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.038017 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:36.471226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.470945 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:36.471226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.471120 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:36.471226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.471181 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:36.613233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.608903 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:36.613233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.609052 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:36.613233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.609096 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:36.732287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.709248 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" event={"ID":"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9","Type":"ContainerStarted","Data":"9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8"} Oct 31 03:29:36.764851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:36.764792 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podStartSLOduration=2.764187559 podStartE2EDuration="2.764187559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:36.741995383 +0000 UTC m=+800.936518505" watchObservedRunningTime="2026-10-31 03:29:36.764187559 +0000 UTC m=+800.958710682" Oct 31 03:29:37.048289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.046627 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.048289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.046690 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:37.149226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.149084 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:37.149402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.149279 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.149402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.149322 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:37.470224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.467653 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.470224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.467715 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:37.610776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.609267 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.610776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:37.609362 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:38.002366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.002313 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.044226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.044041 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.044226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.044103 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:38.136228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.129310 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.151227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.146634 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.151227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.146697 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:38.177373 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.177327 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.178079 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.177621 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.178961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.177659 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.244339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.244295 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.470619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.470583 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.470879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.470852 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:38.617371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.617336 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.617615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.617591 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:38.715511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.714295 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.886311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.874313 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:38.886311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.875749 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:38.901356 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.012687 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.024187 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.052870 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.052946 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.161321 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.162604 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.203326 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.203831 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.204367 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:29:39.241295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.204450 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:29:39.260815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.260781 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.261139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.261108 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:39.276105 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.276069 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.286334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.276974 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:29:39.417226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.397687 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.494895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.493877 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.496133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.496100 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:39.659443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.659406 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.664245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.664186 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:39.734625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.734551 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:29:39.735480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.735456 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:39.764284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.760894 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.822929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.822893 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:29:39.897826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.897779 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.899139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.898805 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:39.899139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.899121 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:29:39.899361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:39.899178 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:29:40.033289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.025299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:29:40.056015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.054809 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.146232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.107364 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.146232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.107429 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:40.147477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.147448 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.147916 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.147695 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:40.169645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.169584 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.216438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.216396 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:29:40.256225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.255959 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.262848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.262810 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.293319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.284521 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.293665 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:29:40.293634 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:29:40.298787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.298758 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:29:40.550290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.547344 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.550290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.547415 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:40.607618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.607580 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.607877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.607847 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:40.830563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.830460 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.831267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.831242 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:40.834525 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.834495 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:29:40.877382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.877341 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.967290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:40.949216 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:41.030445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.030392 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:29:41.041321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.041181 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.042116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.041589 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:41.155654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.155552 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.191228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.185685 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.212231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.192083 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.212553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.212517 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:41.213004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.193727 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:29:41.256250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.256184 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.277371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.277327 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:29:41.340878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.340843 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.396828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.396797 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:41.475664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.475621 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.475938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.475903 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:41.613266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.604268 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.613546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.613509 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:41.841287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.837878 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:42.001265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:41.990708 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:42.041856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.041807 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.042141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.042097 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:42.207286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.168180 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.207286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.191693 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.214836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.214312 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.214836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.214388 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:42.252226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.235518 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:29:42.264395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.263296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.303235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.283292 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.344679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.344634 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:29:42.496323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.488969 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.496323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.489041 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:42.618644 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.618599 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.618902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:42.618877 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:43.008227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.000722 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.036225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.035600 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.036225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.035658 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:43.048246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.043282 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.148725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.148690 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.149001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.148959 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:43.178360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.178297 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.179720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.179688 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:29:43.240230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.239347 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.264394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.264290 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:29:43.287229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.284383 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:29:43.352041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.351319 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.422230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.421320 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.441368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.441323 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.469547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.469512 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.469825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.469801 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:43.605759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.605654 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.606020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.605977 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:43.786431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.785296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.886594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:43.886496 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.004469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.003305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:29:44.038026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.037609 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.038026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.037665 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:44.047261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.045336 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.047261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.045762 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:29:44.156437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.156344 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.157058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.157025 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:44.179270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.179229 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.180248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.180226 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:29:44.185370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.185339 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.243398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.243345 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.250246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.249229 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:29:44.265096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.265048 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:29:44.303185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.302136 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.324686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.324281 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 31 03:29:44.361444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.360334 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.381475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.381445 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:29:44.458780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.458737 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.497226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.496859 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:44.497610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.497514 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:29:44.530766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.530722 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.531031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.531006 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:44.537248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.530854 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:44.609052 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.604882 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.609052 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.604938 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:44.795284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.791661 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:29:44.795284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.792381 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.806677 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.806091 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:29:44.893770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.889825 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:29:44.893770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.891552 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:44.987882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:44.987841 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Oct 31 03:29:45.040636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.040601 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.041151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.041123 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:45.049840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.049710 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.149514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.149478 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:29:45.159302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.159242 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.159555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.159528 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:45.160001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.159966 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.205229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.176250 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:29:45.206377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.194332 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.207181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.194573 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.244681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.244639 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.460340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.460294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.590230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.507275 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j"] Oct 31 03:29:45.679233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.515513 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.679233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.678346 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 03:29:45.713285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.671564 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.713611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.713571 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:45.743356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.531295 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.895226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:45.890298 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.056318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.054420 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:46.056318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.054497 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:46.132935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.132888 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.138985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.138940 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.151312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.151281 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"distinct-2-w1-6c6050-kserve-self-signed-certs\"" Oct 31 03:29:46.152015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.151992 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:46.152173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.152152 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:46.152679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.152655 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:29:46.220227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.219562 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.267143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.254378 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:29:46.267143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.254775 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:29:46.285812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.283977 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.340284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.337540 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:29:46.340284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.337667 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j"] Oct 31 03:29:46.363231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.349890 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:29:46.367651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.367615 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-dshm\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.367914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.367890 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b46c5ba-21f9-4d91-adbc-16e654451c06-tls-certs\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.368148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.368131 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-model-cache\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.368313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.368294 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nflt\" (UniqueName: \"kubernetes.io/projected/3b46c5ba-21f9-4d91-adbc-16e654451c06-kube-api-access-8nflt\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.368518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.368499 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-home\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.380470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.380439 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-tmp-dir\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.491408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.491361 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-model-cache\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.552664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.552621 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nflt\" (UniqueName: \"kubernetes.io/projected/3b46c5ba-21f9-4d91-adbc-16e654451c06-kube-api-access-8nflt\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.553009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.552991 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-home\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.553155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.553139 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-tmp-dir\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.569498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.569464 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-dshm\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.574071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.574040 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b46c5ba-21f9-4d91-adbc-16e654451c06-tls-certs\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.574897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.567686 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-tmp-dir\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.584523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.584480 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.646278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.568509 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-home\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.646278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.496938 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-model-cache\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.646278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.604044 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 03:29:46.646278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.604092 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 03:29:46.646278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.624368 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-dshm\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.679392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.669872 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b46c5ba-21f9-4d91-adbc-16e654451c06-tls-certs\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.679392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.672412 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nflt\" (UniqueName: \"kubernetes.io/projected/3b46c5ba-21f9-4d91-adbc-16e654451c06-kube-api-access-8nflt\") pod \"distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:46.694254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.691104 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:29:46.716230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.702081 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:29:46.748568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.737161 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:29:46.850990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.850411 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:29:46.892409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.890252 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:29:46.907916 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:46.907245 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:47.039602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.035968 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 03:29:47.039602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.036027 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 03:29:47.183556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.177072 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 03:29:47.183556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.177224 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 03:29:47.183556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.178321 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:47.186103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.186063 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:47.189619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.189598 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:29:47.358214 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.358151 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j"] Oct 31 03:29:47.382520 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:29:47.381535 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b46c5ba_21f9_4d91_adbc_16e654451c06.slice/crio-fb72c48f0c6ec2aa55754b28355fd81b132dd1cd429493dfcc7757eb635d4706 WatchSource:0}: Error finding container fb72c48f0c6ec2aa55754b28355fd81b132dd1cd429493dfcc7757eb635d4706: Status 404 returned error can't find the container with id fb72c48f0c6ec2aa55754b28355fd81b132dd1cd429493dfcc7757eb635d4706 Oct 31 03:29:47.689762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:47.689728 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" event={"ID":"3b46c5ba-21f9-4d91-adbc-16e654451c06","Type":"ContainerStarted","Data":"fb72c48f0c6ec2aa55754b28355fd81b132dd1cd429493dfcc7757eb635d4706"} Oct 31 03:29:48.593224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.592769 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:48.600247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.594827 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:48.600247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.598364 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:29:48.640224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.633153 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:29:48.640224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.638497 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:48.734160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.734113 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" event={"ID":"3b46c5ba-21f9-4d91-adbc-16e654451c06","Type":"ContainerStarted","Data":"945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3"} Oct 31 03:29:48.770224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.759290 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podStartSLOduration=3.759269997 podStartE2EDuration="3.759269997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:48.758569112 +0000 UTC m=+812.953092255" watchObservedRunningTime="2026-10-31 03:29:48.759269997 +0000 UTC m=+812.953793119" Oct 31 03:29:48.928009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:48.927920 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:49.204228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.203731 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:29:49.204228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.203789 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:49.395434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.394560 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:49.395677 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.395654 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:49.396383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.396363 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:29:49.520808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.520712 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:49.521550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.521530 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:49.522839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:49.522820 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:29:56.908411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:56.908322 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:56.922089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:56.922056 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:58.024227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:58.019831 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:29:59.266364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:59.266315 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.294229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:59.289299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.735705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:59.735650 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.802230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:59.801366 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.885229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:59.884350 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.978251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:29:59.969324 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.026439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.026323 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.154241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.151571 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:00.154474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.154283 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.177353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.177270 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g"] Oct 31 03:30:00.202238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.198973 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:00.207244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.207195 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:30:00.271187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.271125 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.359132 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.281695 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr"] Oct 31 03:30:00.654702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.286182 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c507592e-7a1a-4634-88a1-59ddf949c1dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:00.654702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.607586 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2pr\" (UniqueName: \"kubernetes.io/projected/c507592e-7a1a-4634-88a1-59ddf949c1dd-kube-api-access-sq2pr\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:00.654702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.354315 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.654702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.354351 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:00.654702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.299958 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.713370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.713327 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c507592e-7a1a-4634-88a1-59ddf949c1dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:00.713680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.713657 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2pr\" (UniqueName: \"kubernetes.io/projected/c507592e-7a1a-4634-88a1-59ddf949c1dd-kube-api-access-sq2pr\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:00.715845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.715816 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c507592e-7a1a-4634-88a1-59ddf949c1dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:00.954696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:00.908388 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:01.119505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.119466 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g"] Oct 31 03:30:01.332967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.332880 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr"] Oct 31 03:30:01.411478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.411438 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:30:01.411884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.411863 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr"] Oct 31 03:30:01.427226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.295530 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.456227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.277324 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.456227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.306422 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81ff07-0ad2-454f-b6cb-ed7663288119-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-sqbfr\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:01.456227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.451984 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7cx\" (UniqueName: \"kubernetes.io/projected/8e81ff07-0ad2-454f-b6cb-ed7663288119-kube-api-access-ql7cx\") pod \"maas-api-key-cleanup-29890290-sqbfr\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:01.456227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.145432 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:01.459406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.383665 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:30:01.516227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.296012 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:30:01.594282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.566994 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7cx\" (UniqueName: \"kubernetes.io/projected/8e81ff07-0ad2-454f-b6cb-ed7663288119-kube-api-access-ql7cx\") pod \"maas-api-key-cleanup-29890290-sqbfr\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:01.635405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.592781 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2pr\" (UniqueName: \"kubernetes.io/projected/c507592e-7a1a-4634-88a1-59ddf949c1dd-kube-api-access-sq2pr\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:01.635405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.631531 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.643010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.642964 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.663840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.663511 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.715783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.715728 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr"] Oct 31 03:30:01.722457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.722420 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:30:01.722763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.722749 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:01.740327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.722904 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:30:01.758319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.758290 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:30:01.758637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.758619 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:30:01.796446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.796415 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:01.797347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.741567 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:01.820721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.778721 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81ff07-0ad2-454f-b6cb-ed7663288119-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-sqbfr\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:01.821732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.759579 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7cx\" (UniqueName: \"kubernetes.io/projected/8e81ff07-0ad2-454f-b6cb-ed7663288119-kube-api-access-ql7cx\") pod \"maas-api-key-cleanup-29890290-sqbfr\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:01.865780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.840883 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:01.866084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.860435 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.914227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.903660 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81ff07-0ad2-454f-b6cb-ed7663288119-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-sqbfr\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:01.922475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:01.918281 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.006228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.005270 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:30:02.176281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.167271 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:02.176281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.168052 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f739d2c-54df-4030-90b6-9d30a03e62d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:02.176281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.175561 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpv5r\" (UniqueName: \"kubernetes.io/projected/9f739d2c-54df-4030-90b6-9d30a03e62d1-kube-api-access-wpv5r\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:02.202224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.180423 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:02.300303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.285634 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpv5r\" (UniqueName: \"kubernetes.io/projected/9f739d2c-54df-4030-90b6-9d30a03e62d1-kube-api-access-wpv5r\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:02.300303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.285762 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f739d2c-54df-4030-90b6-9d30a03e62d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:02.301182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.301132 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.303246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.302337 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.450355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.425514 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:30:02.450355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.426241 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:30:02.458429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.458400 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpv5r\" (UniqueName: \"kubernetes.io/projected/9f739d2c-54df-4030-90b6-9d30a03e62d1-kube-api-access-wpv5r\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:02.500075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.500032 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.537231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.503936 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:02.537231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.526151 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.571227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.538089 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f739d2c-54df-4030-90b6-9d30a03e62d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:02.600410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.600361 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:02.604266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.600674 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.619426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.619392 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:02.657422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.646427 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:30:02.657422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.646519 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:02.705759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.705448 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:02.732157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.732096 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.881241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.826604 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:30:02.881241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.826694 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.898268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.705555 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:30:02.898860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.898819 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.941298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.901278 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:30:02.941799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:02.941773 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:30:03.040349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.014743 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:30:03.097905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.097875 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:30:03.283462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.283414 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:03.283784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.283737 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.314530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.314426 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.316869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.316836 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:30:03.351228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.342049 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.376767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.364325 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:30:03.462130 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.461414 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.506653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.506615 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:30:03.534363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.534315 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.666286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.646956 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.721626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.664300 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.745659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.745598 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.746305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.746281 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:30:03.863244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.841789 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.863244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.842243 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:03.991559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.949316 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.991559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:03.950788 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:04.083979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.083946 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:30:04.084365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.084348 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:30:04.084564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.084551 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:30:04.088536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.088521 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:04.100343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.100298 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:04.159787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.146290 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:30:04.159787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.156466 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:30:04.320279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.315277 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:30:04.344575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.344252 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 31 03:30:04.407379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.400267 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:04.515275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.513266 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:04.711569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.695108 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj"] Oct 31 03:30:04.726974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.725668 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:04.774082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.765583 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj"] Oct 31 03:30:04.774082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.765746 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.800736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.797277 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:30:04.804248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.801020 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:04.809750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.809729 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:30:04.834320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.834281 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:30:04.835736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.835707 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:30:04.846359 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.846295 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:30:04.908973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.908936 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910421 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910476 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910526 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910604 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910641 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910672 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2xt\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-kube-api-access-bk2xt\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910850 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910813 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.910906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.910867 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:04.955852 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.955826 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr"] Oct 31 03:30:04.958724 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:30:04.958693 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e81ff07_0ad2_454f_b6cb_ed7663288119.slice/crio-1c3a07163b2acfcc2841b77dd47a51ce60fb6eaa7b83cc2f319246374adc624a WatchSource:0}: Error finding container 1c3a07163b2acfcc2841b77dd47a51ce60fb6eaa7b83cc2f319246374adc624a: Status 404 returned error can't find the container with id 1c3a07163b2acfcc2841b77dd47a51ce60fb6eaa7b83cc2f319246374adc624a Oct 31 03:30:04.965848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.965825 5520 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:30:04.977093 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.975800 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr"] Oct 31 03:30:04.977093 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:04.976925 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" event={"ID":"8e81ff07-0ad2-454f-b6cb-ed7663288119","Type":"ContainerStarted","Data":"1c3a07163b2acfcc2841b77dd47a51ce60fb6eaa7b83cc2f319246374adc624a"} Oct 31 03:30:04.979745 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:30:04.979721 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f739d2c_54df_4030_90b6_9d30a03e62d1.slice/crio-d05c13920b73f78bc2d952656e3c710fb937921bd6a77168c51ac2d984ee63d8 WatchSource:0}: Error finding container d05c13920b73f78bc2d952656e3c710fb937921bd6a77168c51ac2d984ee63d8: Status 404 returned error can't find the container with id d05c13920b73f78bc2d952656e3c710fb937921bd6a77168c51ac2d984ee63d8 Oct 31 03:30:05.012400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012367 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2xt\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-kube-api-access-bk2xt\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.012590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012495 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.012590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012533 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.012718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012704 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.012860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012834 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.012920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012856 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.012920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012889 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.013012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012945 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.013012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.012972 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.013414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.013392 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.013967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.013943 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.014599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.014287 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.014599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.014551 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.014914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.014884 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.017759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.017732 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.018355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.018327 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.022464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.022440 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2xt\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-kube-api-access-bk2xt\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.022554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.022502 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.053442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.053417 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g"] Oct 31 03:30:05.055041 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:30:05.055002 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc507592e_7a1a_4634_88a1_59ddf949c1dd.slice/crio-e4c5986494d430815b1c71a354ac76e4783f56e568359d0e488bce835e0e316f WatchSource:0}: Error finding container e4c5986494d430815b1c71a354ac76e4783f56e568359d0e488bce835e0e316f: Status 404 returned error can't find the container with id e4c5986494d430815b1c71a354ac76e4783f56e568359d0e488bce835e0e316f Oct 31 03:30:05.127704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.127617 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:05.426496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.426438 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj"] Oct 31 03:30:05.429453 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:30:05.429421 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec9a673_d7aa_4b7d_807e_d519c2a29e2c.slice/crio-816d4ff9711500252b731782513adaf479b7de32dc2a762ddd723e707dd3001b WatchSource:0}: Error finding container 816d4ff9711500252b731782513adaf479b7de32dc2a762ddd723e707dd3001b: Status 404 returned error can't find the container with id 816d4ff9711500252b731782513adaf479b7de32dc2a762ddd723e707dd3001b Oct 31 03:30:05.432173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.432123 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:05.432275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.432241 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:05.432316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.432286 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:05.985290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.985254 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" event={"ID":"9f739d2c-54df-4030-90b6-9d30a03e62d1","Type":"ContainerStarted","Data":"d05c13920b73f78bc2d952656e3c710fb937921bd6a77168c51ac2d984ee63d8"} Oct 31 03:30:05.986721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.986624 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" event={"ID":"c507592e-7a1a-4634-88a1-59ddf949c1dd","Type":"ContainerStarted","Data":"e4c5986494d430815b1c71a354ac76e4783f56e568359d0e488bce835e0e316f"} Oct 31 03:30:05.988678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.988656 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" event={"ID":"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c","Type":"ContainerStarted","Data":"f13606988022784e984b0ad1a5e07c992fa1bf0f995ce6ad3ddb98488786ede4"} Oct 31 03:30:05.988776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:05.988695 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" event={"ID":"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c","Type":"ContainerStarted","Data":"816d4ff9711500252b731782513adaf479b7de32dc2a762ddd723e707dd3001b"} Oct 31 03:30:06.128532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:06.128493 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:06.129840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:06.129799 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:30:06.130050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:06.130025 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:30:07.136224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:07.134573 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:30:07.136224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:07.134649 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:30:08.132240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:08.128610 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:30:08.132240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:08.128672 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:30:09.128846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:09.128366 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:30:09.128846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:09.128424 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:30:09.417268 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:09.416781 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:30:10.128743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:10.128707 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:30:10.128991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:10.128961 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:30:11.129296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:11.129258 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:30:11.129701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:11.129332 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:30:12.128141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.128103 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:30:12.128390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.128164 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:30:12.148230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.147971 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" event={"ID":"8e81ff07-0ad2-454f-b6cb-ed7663288119","Type":"ContainerStarted","Data":"a149a2b5e53edda907051d3127284f1a9c5cb7b3241b9c69f2847387b68ba6b4"} Oct 31 03:30:12.152620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.152592 5520 generic.go:356] "Generic (PLEG): container finished" podID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerID="ba0655eefa07de3206e77a69c8db17c989a1323df3ca6b9551a12325310d394e" exitCode=60 Oct 31 03:30:12.152820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.152801 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" event={"ID":"c507592e-7a1a-4634-88a1-59ddf949c1dd","Type":"ContainerDied","Data":"ba0655eefa07de3206e77a69c8db17c989a1323df3ca6b9551a12325310d394e"} Oct 31 03:30:12.153269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.153244 5520 scope.go:122] "RemoveContainer" containerID="ba0655eefa07de3206e77a69c8db17c989a1323df3ca6b9551a12325310d394e" Oct 31 03:30:12.166965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.166892 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podStartSLOduration=8.166853144 podStartE2EDuration="8.166853144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.017789961 +0000 UTC m=+830.212313083" watchObservedRunningTime="2026-10-31 03:30:12.166853144 +0000 UTC m=+836.361376264" Oct 31 03:30:12.170408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:12.170359 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" podStartSLOduration=5.696541228 podStartE2EDuration="12.170348096s" totalImagesPullingTime="6.473806868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:12.1639136 +0000 UTC m=+836.358436722" watchObservedRunningTime="2026-10-31 03:30:12.170348096 +0000 UTC m=+836.364871219" Oct 31 03:30:13.161444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.161358 5520 generic.go:356] "Generic (PLEG): container finished" podID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerID="80f8a8ad4deced679154e4d3db3abefac0aa40c31505683e097f6b81e6ed9736" exitCode=60 Oct 31 03:30:13.161444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.161407 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" event={"ID":"9f739d2c-54df-4030-90b6-9d30a03e62d1","Type":"ContainerDied","Data":"80f8a8ad4deced679154e4d3db3abefac0aa40c31505683e097f6b81e6ed9736"} Oct 31 03:30:13.161894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.161800 5520 scope.go:122] "RemoveContainer" containerID="80f8a8ad4deced679154e4d3db3abefac0aa40c31505683e097f6b81e6ed9736" Oct 31 03:30:13.163631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.163542 5520 generic.go:356] "Generic (PLEG): container finished" podID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerID="a149a2b5e53edda907051d3127284f1a9c5cb7b3241b9c69f2847387b68ba6b4" exitCode=60 Oct 31 03:30:13.163631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.163572 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" event={"ID":"8e81ff07-0ad2-454f-b6cb-ed7663288119","Type":"ContainerDied","Data":"a149a2b5e53edda907051d3127284f1a9c5cb7b3241b9c69f2847387b68ba6b4"} Oct 31 03:30:13.163933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.163917 5520 scope.go:122] "RemoveContainer" containerID="a149a2b5e53edda907051d3127284f1a9c5cb7b3241b9c69f2847387b68ba6b4" Oct 31 03:30:13.166439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.166420 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" event={"ID":"c507592e-7a1a-4634-88a1-59ddf949c1dd","Type":"ContainerStarted","Data":"555f3ad8e49f35c1f32b9d215d84365d74860071b95e58cec7c742504fc94271"} Oct 31 03:30:13.206021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.205961 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" podStartSLOduration=7.253667743 podStartE2EDuration="13.205938314s" totalImagesPullingTime="5.952270571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:13.199521211 +0000 UTC m=+837.394044333" watchObservedRunningTime="2026-10-31 03:30:13.205938314 +0000 UTC m=+837.400461435" Oct 31 03:30:13.268841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.268807 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj"] Oct 31 03:30:13.274337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.274318 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.285234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.285181 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj"] Oct 31 03:30:13.286154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.286131 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:30:13.369232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.369177 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bd4z\" (UniqueName: \"kubernetes.io/projected/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-kube-api-access-6bd4z\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.369427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.369362 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.369427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.369393 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.404234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.395591 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:13.404234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.397350 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:13.404601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.404487 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:30:13.472226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.472094 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bd4z\" (UniqueName: \"kubernetes.io/projected/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-kube-api-access-6bd4z\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.472420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.472257 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.472420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.472303 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.482285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.476533 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.482285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.477244 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.493228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.489844 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bd4z\" (UniqueName: \"kubernetes.io/projected/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-kube-api-access-6bd4z\") pod \"maas-api-revoke-sub-0e7b20be-7ktlj\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.613724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.613688 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:13.899935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:13.899912 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj"] Oct 31 03:30:13.901891 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:30:13.901862 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abc4cd2_1bc2_4cf6_b344_ba4d2fb277bd.slice/crio-30b1aee6dcaa19b5f168a4ead84c67f19e6e0579391447e7ff78c38d0bd3b081 WatchSource:0}: Error finding container 30b1aee6dcaa19b5f168a4ead84c67f19e6e0579391447e7ff78c38d0bd3b081: Status 404 returned error can't find the container with id 30b1aee6dcaa19b5f168a4ead84c67f19e6e0579391447e7ff78c38d0bd3b081 Oct 31 03:30:14.221507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.221472 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" event={"ID":"9f739d2c-54df-4030-90b6-9d30a03e62d1","Type":"ContainerStarted","Data":"4bdc0c6c4323a3029d973bdbb977bba6ac564d26cebd836b1ebefe1d051a4a1e"} Oct 31 03:30:14.225025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.224972 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" event={"ID":"8e81ff07-0ad2-454f-b6cb-ed7663288119","Type":"ContainerStarted","Data":"26c3b6b138f4ac29aed85a86d594ab595bcda0ff4541f2770f5dc34da205137b"} Oct 31 03:30:14.227794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.227761 5520 generic.go:356] "Generic (PLEG): container finished" podID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerID="555f3ad8e49f35c1f32b9d215d84365d74860071b95e58cec7c742504fc94271" exitCode=60 Oct 31 03:30:14.227906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.227833 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" event={"ID":"c507592e-7a1a-4634-88a1-59ddf949c1dd","Type":"ContainerDied","Data":"555f3ad8e49f35c1f32b9d215d84365d74860071b95e58cec7c742504fc94271"} Oct 31 03:30:14.227906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.227872 5520 scope.go:122] "RemoveContainer" containerID="ba0655eefa07de3206e77a69c8db17c989a1323df3ca6b9551a12325310d394e" Oct 31 03:30:14.228269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.228249 5520 scope.go:122] "RemoveContainer" containerID="555f3ad8e49f35c1f32b9d215d84365d74860071b95e58cec7c742504fc94271" Oct 31 03:30:14.228562 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:14.228536 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g_odh-ai-gateway-infra(c507592e-7a1a-4634-88a1-59ddf949c1dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" Oct 31 03:30:14.229927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.229902 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" event={"ID":"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd","Type":"ContainerStarted","Data":"30b1aee6dcaa19b5f168a4ead84c67f19e6e0579391447e7ff78c38d0bd3b081"} Oct 31 03:30:14.243348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:14.243306 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" podStartSLOduration=7.325882275 podStartE2EDuration="14.243293317s" totalImagesPullingTime="6.917411042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:14.241906051 +0000 UTC m=+838.436429308" watchObservedRunningTime="2026-10-31 03:30:14.243293317 +0000 UTC m=+838.437816443" Oct 31 03:30:15.239015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.238936 5520 generic.go:356] "Generic (PLEG): container finished" podID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerID="26c3b6b138f4ac29aed85a86d594ab595bcda0ff4541f2770f5dc34da205137b" exitCode=60 Oct 31 03:30:15.239528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.239024 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" event={"ID":"8e81ff07-0ad2-454f-b6cb-ed7663288119","Type":"ContainerDied","Data":"26c3b6b138f4ac29aed85a86d594ab595bcda0ff4541f2770f5dc34da205137b"} Oct 31 03:30:15.239528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.239096 5520 scope.go:122] "RemoveContainer" containerID="a149a2b5e53edda907051d3127284f1a9c5cb7b3241b9c69f2847387b68ba6b4" Oct 31 03:30:15.239528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.239497 5520 scope.go:122] "RemoveContainer" containerID="26c3b6b138f4ac29aed85a86d594ab595bcda0ff4541f2770f5dc34da205137b" Oct 31 03:30:15.239894 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:15.239872 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-sqbfr_odh-ai-gateway-infra(8e81ff07-0ad2-454f-b6cb-ed7663288119)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" Oct 31 03:30:15.242218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.242179 5520 scope.go:122] "RemoveContainer" containerID="555f3ad8e49f35c1f32b9d215d84365d74860071b95e58cec7c742504fc94271" Oct 31 03:30:15.242554 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:15.242531 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g_odh-ai-gateway-infra(c507592e-7a1a-4634-88a1-59ddf949c1dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" Oct 31 03:30:15.243665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.243641 5520 generic.go:356] "Generic (PLEG): container finished" podID="3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" containerID="e16a69508e84f56a4aaf9527bb336b380e1ca275ee9152f17e8130987bb57c74" exitCode=0 Oct 31 03:30:15.243759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.243718 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" event={"ID":"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd","Type":"ContainerDied","Data":"e16a69508e84f56a4aaf9527bb336b380e1ca275ee9152f17e8130987bb57c74"} Oct 31 03:30:15.245719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.245698 5520 generic.go:356] "Generic (PLEG): container finished" podID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerID="4bdc0c6c4323a3029d973bdbb977bba6ac564d26cebd836b1ebefe1d051a4a1e" exitCode=60 Oct 31 03:30:15.245812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.245785 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" event={"ID":"9f739d2c-54df-4030-90b6-9d30a03e62d1","Type":"ContainerDied","Data":"4bdc0c6c4323a3029d973bdbb977bba6ac564d26cebd836b1ebefe1d051a4a1e"} Oct 31 03:30:15.246111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.246094 5520 scope.go:122] "RemoveContainer" containerID="4bdc0c6c4323a3029d973bdbb977bba6ac564d26cebd836b1ebefe1d051a4a1e" Oct 31 03:30:15.246447 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:15.246428 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr_odh-ai-gateway-infra(9f739d2c-54df-4030-90b6-9d30a03e62d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" Oct 31 03:30:15.252572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:15.252556 5520 scope.go:122] "RemoveContainer" containerID="80f8a8ad4deced679154e4d3db3abefac0aa40c31505683e097f6b81e6ed9736" Oct 31 03:30:16.267130 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.267093 5520 scope.go:122] "RemoveContainer" containerID="4bdc0c6c4323a3029d973bdbb977bba6ac564d26cebd836b1ebefe1d051a4a1e" Oct 31 03:30:16.267749 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:16.267475 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr_odh-ai-gateway-infra(9f739d2c-54df-4030-90b6-9d30a03e62d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" Oct 31 03:30:16.268909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.268888 5520 scope.go:122] "RemoveContainer" containerID="26c3b6b138f4ac29aed85a86d594ab595bcda0ff4541f2770f5dc34da205137b" Oct 31 03:30:16.269417 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:16.269185 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-sqbfr_odh-ai-gateway-infra(8e81ff07-0ad2-454f-b6cb-ed7663288119)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" Oct 31 03:30:16.441576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.441547 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:16.641813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.641731 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-kube-api-access-6bd4z\" (UniqueName: \"kubernetes.io/projected/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-kube-api-access-6bd4z\") pod \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " Oct 31 03:30:16.641813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.641806 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-openshift-ca\") pod \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " Oct 31 03:30:16.642054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.641944 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-xks-ca\") pod \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\" (UID: \"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd\") " Oct 31 03:30:16.642181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.642152 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-openshift-ca" pod "3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" (UID: "3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:16.642628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.642608 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:16.644196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.644169 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-xks-ca" pod "3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" (UID: "3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:16.644196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.644184 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-kube-api-access-6bd4z" pod "3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" (UID: "3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd"). InnerVolumeSpecName "kube-api-access-6bd4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:16.743690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.743658 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bd4z\" (UniqueName: \"kubernetes.io/projected/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-kube-api-access-6bd4z\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:16.743690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:16.743688 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:17.277230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:17.277186 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" Oct 31 03:30:17.277230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:17.277223 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj" event={"ID":"3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd","Type":"ContainerDied","Data":"30b1aee6dcaa19b5f168a4ead84c67f19e6e0579391447e7ff78c38d0bd3b081"} Oct 31 03:30:17.277729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:17.277275 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30b1aee6dcaa19b5f168a4ead84c67f19e6e0579391447e7ff78c38d0bd3b081" Oct 31 03:30:19.253225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.245312 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d"] Oct 31 03:30:19.253225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.246302 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" containerName="revoke-keys" Oct 31 03:30:19.253225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.246325 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" containerName="revoke-keys" Oct 31 03:30:19.253225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.246545 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" containerName="revoke-keys" Oct 31 03:30:19.294612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.293797 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d"] Oct 31 03:30:19.294612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.293935 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385086 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385136 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385293 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385331 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385532 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385634 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385699 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385734 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.385817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.385795 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n66k5\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-kube-api-access-n66k5\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.487555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487521 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.487721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487596 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.487721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487655 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.487721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487681 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.487866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487735 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n66k5\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-kube-api-access-n66k5\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.487866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487777 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.487866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487805 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.488012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487908 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.488012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487948 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.488012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.487954 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.488155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.488009 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.488371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.488341 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.488485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.488369 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.488944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.488920 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.490438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.490414 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.490573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.490548 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.495958 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.495931 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.495958 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.495949 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n66k5\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-kube-api-access-n66k5\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.543759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.543684 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll"] Oct 31 03:30:19.584590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.584561 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll"] Oct 31 03:30:19.584778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.584687 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.608432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.608405 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:19.716658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.716614 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.716844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.716707 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.716844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.716781 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.716944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.716927 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.716980 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.716971 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.717023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.717009 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42611746-9ded-4093-8e03-087bf1734ab3-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.717058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.717047 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfss9\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-kube-api-access-dfss9\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.717144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.717131 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42611746-9ded-4093-8e03-087bf1734ab3-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.717185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.717160 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.818986 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42611746-9ded-4093-8e03-087bf1734ab3-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819027 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfss9\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-kube-api-access-dfss9\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819083 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42611746-9ded-4093-8e03-087bf1734ab3-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819100 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819224 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819276 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819330 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819482 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819512 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819609 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819721 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.819854 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.819814 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42611746-9ded-4093-8e03-087bf1734ab3-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.820028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.820007 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.820113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.820041 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.822119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.822082 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.822257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.822185 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42611746-9ded-4093-8e03-087bf1734ab3-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.827098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.827077 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.827333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.827312 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfss9\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-kube-api-access-dfss9\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.891071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.891040 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt"] Oct 31 03:30:19.896825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.896801 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:19.901469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.901441 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:19.913310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.909323 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt"] Oct 31 03:30:19.935636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.935593 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:19.935866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.935685 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:19.935866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.935729 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:19.938306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:19.938286 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d"] Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034422 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6gl\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-kube-api-access-4d6gl\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034473 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034503 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034542 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034591 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034719 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034916 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034961 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.035225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.034993 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.168908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.168813 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.172557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.172531 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.173120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.173098 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.173680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.173657 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.175432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.175409 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6gl\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-kube-api-access-4d6gl\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.175895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.175877 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.176380 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.176439 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.176505 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.177030 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.174821 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.177288 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.172232 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.177979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.173020 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.188642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.188571 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.192515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.192457 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.197396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.197261 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6gl\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-kube-api-access-4d6gl\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.197988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.197934 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.230802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.230601 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:20.258919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.258874 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll"] Oct 31 03:30:20.259875 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:30:20.259819 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42611746_9ded_4093_8e03_087bf1734ab3.slice/crio-29d752c74d92ff0b6e321cb8fffa6d5a8d326c6b2bca173614ba072a53700aa0 WatchSource:0}: Error finding container 29d752c74d92ff0b6e321cb8fffa6d5a8d326c6b2bca173614ba072a53700aa0: Status 404 returned error can't find the container with id 29d752c74d92ff0b6e321cb8fffa6d5a8d326c6b2bca173614ba072a53700aa0 Oct 31 03:30:20.263296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.263263 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:20.263391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.263342 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:20.263447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.263391 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:20.311742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.311684 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" event={"ID":"42611746-9ded-4093-8e03-087bf1734ab3","Type":"ContainerStarted","Data":"29d752c74d92ff0b6e321cb8fffa6d5a8d326c6b2bca173614ba072a53700aa0"} Oct 31 03:30:20.314427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.314390 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" event={"ID":"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd","Type":"ContainerStarted","Data":"13c9dbb17fbd0fcd409b96501b0e5240552dcf791420161c40914cfe807013f2"} Oct 31 03:30:20.314595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.314580 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" event={"ID":"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd","Type":"ContainerStarted","Data":"59d6f3456c50d5be6c01ff9a3bdcceb9b45c896d96c4ff20ee80fb1c608c3448"} Oct 31 03:30:20.351928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.351796 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podStartSLOduration=1.351783565 podStartE2EDuration="1.351783565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:20.349740471 +0000 UTC m=+844.544263612" watchObservedRunningTime="2026-10-31 03:30:20.351783565 +0000 UTC m=+844.546306683" Oct 31 03:30:20.540822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.540788 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt"] Oct 31 03:30:20.543064 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:30:20.543032 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a8ffdd_6cda_4f06_aaa6_edaf38a7dab7.slice/crio-54ddcaf10c4b08bb0eaf803c1115e33de306b46bda86b25b435c293f43bb5ef5 WatchSource:0}: Error finding container 54ddcaf10c4b08bb0eaf803c1115e33de306b46bda86b25b435c293f43bb5ef5: Status 404 returned error can't find the container with id 54ddcaf10c4b08bb0eaf803c1115e33de306b46bda86b25b435c293f43bb5ef5 Oct 31 03:30:20.546148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.546111 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:20.546275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.546195 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:20.546275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.546255 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:30:20.609313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:20.609284 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:21.339351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.339314 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" event={"ID":"42611746-9ded-4093-8e03-087bf1734ab3","Type":"ContainerStarted","Data":"55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d"} Oct 31 03:30:21.341541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.341515 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" event={"ID":"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7","Type":"ContainerStarted","Data":"c456f6196a172ad408c109fe2bcdb0bfc7e913a721b008bb49c03888322873a2"} Oct 31 03:30:21.341637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.341556 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" event={"ID":"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7","Type":"ContainerStarted","Data":"54ddcaf10c4b08bb0eaf803c1115e33de306b46bda86b25b435c293f43bb5ef5"} Oct 31 03:30:21.363923 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.363869 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podStartSLOduration=2.363851457 podStartE2EDuration="2.363851457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:21.359773467 +0000 UTC m=+845.554296587" watchObservedRunningTime="2026-10-31 03:30:21.363851457 +0000 UTC m=+845.558374636" Oct 31 03:30:21.404269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.401390 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podStartSLOduration=2.401368129 podStartE2EDuration="2.401368129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:21.386464532 +0000 UTC m=+845.580987653" watchObservedRunningTime="2026-10-31 03:30:21.401368129 +0000 UTC m=+845.595891252" Oct 31 03:30:21.611155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.611015 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:21.611155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.611085 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:21.909311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.907973 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:21.909311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.908117 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:21.909311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:21.908162 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:22.233227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:22.232968 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:22.238224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:22.234342 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:30:22.238224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:22.234397 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 03:30:22.610279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:22.609702 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:22.610279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:22.609774 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:22.905706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:22.905612 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:22.905941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:22.905914 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:23.235230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:23.234343 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:30:23.235230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:23.234470 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 03:30:23.634288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:23.630416 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:23.634288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:23.630527 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:23.962291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:23.912333 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:23.962291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:23.912399 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:24.262284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.256748 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.262284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.256806 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 03:30:24.327467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.327425 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.579284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.568607 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.579284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.570912 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:24.579284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.571740 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:30:24.620314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.620162 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.620314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.620253 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:24.668008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.660689 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.808366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.808320 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:24.895148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.895038 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:30:24.934241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.896991 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.964986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.905381 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:30:24.986226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.906575 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.986499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.986462 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:24.998115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:24.909889 5520 scope.go:122] "RemoveContainer" containerID="555f3ad8e49f35c1f32b9d215d84365d74860071b95e58cec7c742504fc94271" Oct 31 03:30:25.017554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.015280 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:30:25.052744 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.052609 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.061399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.061314 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.099227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.098688 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.141228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.137305 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.234232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.233737 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:30:25.234232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.233814 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 03:30:25.363228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.359411 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.570396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.570353 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.571393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.571368 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:30:25.585228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.573095 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.585794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.581326 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:25.608229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.598241 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:30:25.608229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.599320 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:30:25.618111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.617709 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:25.618111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.617765 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:25.722339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.713871 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.879225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.871480 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.891363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.891319 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.892061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.892038 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:30:25.940230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.935192 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:25.940230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.935275 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:25.967339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.963350 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.997475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:25.997440 5520 scope.go:122] "RemoveContainer" containerID="26c3b6b138f4ac29aed85a86d594ab595bcda0ff4541f2770f5dc34da205137b" Oct 31 03:30:26.095683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.058485 5520 scope.go:122] "RemoveContainer" containerID="4bdc0c6c4323a3029d973bdbb977bba6ac564d26cebd836b1ebefe1d051a4a1e" Oct 31 03:30:26.095683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.058755 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:30:26.324292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.275055 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:30:26.455498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.455461 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.503261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.503231 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:30:26.555442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.555411 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:30:26.592492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.586308 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:30:26.678038 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:26.677986 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc507592e_7a1a_4634_88a1_59ddf949c1dd.slice/crio-29d62d098cdc170851d903c5c02e5411ecadf802c84c3e10e9c3d03e42ed1db7.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:30:26.703273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.703238 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:30:26.726244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.724732 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.726244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.725118 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 03:30:26.731324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.730435 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:26.747593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.745128 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:26.747593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.745182 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:26.752495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.725189 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.752495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.750158 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.831229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.804477 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.857426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.857329 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.963253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.944519 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.964623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.964567 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.989050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:26.988992 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.064250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.052303 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.064250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.052366 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:27.109497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.109409 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:30:27.275973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.275930 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.459055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.458996 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.466967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.466940 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.467293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.467265 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 03:30:27.593750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.593718 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:30:27.597391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.597365 5520 generic.go:356] "Generic (PLEG): container finished" podID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerID="29d62d098cdc170851d903c5c02e5411ecadf802c84c3e10e9c3d03e42ed1db7" exitCode=60 Oct 31 03:30:27.598724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.598700 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" event={"ID":"c507592e-7a1a-4634-88a1-59ddf949c1dd","Type":"ContainerDied","Data":"29d62d098cdc170851d903c5c02e5411ecadf802c84c3e10e9c3d03e42ed1db7"} Oct 31 03:30:27.598934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.598918 5520 scope.go:122] "RemoveContainer" containerID="555f3ad8e49f35c1f32b9d215d84365d74860071b95e58cec7c742504fc94271" Oct 31 03:30:27.599649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.599634 5520 scope.go:122] "RemoveContainer" containerID="29d62d098cdc170851d903c5c02e5411ecadf802c84c3e10e9c3d03e42ed1db7" Oct 31 03:30:27.600436 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:27.600412 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g_odh-ai-gateway-infra(c507592e-7a1a-4634-88a1-59ddf949c1dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" Oct 31 03:30:27.625415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.625384 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.625680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.625644 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:27.726549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.726520 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:30:27.745708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.744242 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.750092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.749486 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.755031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.755010 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:30:27.767504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.767479 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:30:27.836251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.836181 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:30:27.837134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.837110 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:27.840339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.839762 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:30:27.851222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.846489 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:30:27.908231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.908075 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:27.922504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.922427 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.930764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:27.926308 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:28.238299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.237833 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.238299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.237892 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 03:30:28.613527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.613376 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.615703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.613738 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 03:30:28.619586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.619558 5520 generic.go:356] "Generic (PLEG): container finished" podID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerID="c8a7037acb66d1009b6554ef0d6ee6c3d5ecc55d1b8d885afbad98ebb77347de" exitCode=60 Oct 31 03:30:28.619905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.619882 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" event={"ID":"9f739d2c-54df-4030-90b6-9d30a03e62d1","Type":"ContainerDied","Data":"c8a7037acb66d1009b6554ef0d6ee6c3d5ecc55d1b8d885afbad98ebb77347de"} Oct 31 03:30:28.620076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.620062 5520 scope.go:122] "RemoveContainer" containerID="4bdc0c6c4323a3029d973bdbb977bba6ac564d26cebd836b1ebefe1d051a4a1e" Oct 31 03:30:28.620735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.620716 5520 scope.go:122] "RemoveContainer" containerID="c8a7037acb66d1009b6554ef0d6ee6c3d5ecc55d1b8d885afbad98ebb77347de" Oct 31 03:30:28.621300 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:28.621276 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr_odh-ai-gateway-infra(9f739d2c-54df-4030-90b6-9d30a03e62d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" Oct 31 03:30:28.631932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.631893 5520 generic.go:356] "Generic (PLEG): container finished" podID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerID="cac61d88cb302808bbc63c00e1f3c3e235353e1d6a06a4f868b571ec23f8dc69" exitCode=60 Oct 31 03:30:28.632177 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.632157 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" event={"ID":"8e81ff07-0ad2-454f-b6cb-ed7663288119","Type":"ContainerDied","Data":"cac61d88cb302808bbc63c00e1f3c3e235353e1d6a06a4f868b571ec23f8dc69"} Oct 31 03:30:28.632812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.632795 5520 scope.go:122] "RemoveContainer" containerID="cac61d88cb302808bbc63c00e1f3c3e235353e1d6a06a4f868b571ec23f8dc69" Oct 31 03:30:28.633705 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:28.633678 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-sqbfr_odh-ai-gateway-infra(8e81ff07-0ad2-454f-b6cb-ed7663288119)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" Oct 31 03:30:28.656712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.654282 5520 scope.go:122] "RemoveContainer" containerID="26c3b6b138f4ac29aed85a86d594ab595bcda0ff4541f2770f5dc34da205137b" Oct 31 03:30:28.777288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.777238 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g"] Oct 31 03:30:28.916284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.902524 5520 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.916284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:28.902603 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 03:30:29.544826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.534649 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:29.665080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.665038 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c507592e-7a1a-4634-88a1-59ddf949c1dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c507592e-7a1a-4634-88a1-59ddf949c1dd-maas-api-service-ca\") pod \"c507592e-7a1a-4634-88a1-59ddf949c1dd\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " Oct 31 03:30:29.665301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.665109 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c507592e-7a1a-4634-88a1-59ddf949c1dd-kube-api-access-sq2pr\" (UniqueName: \"kubernetes.io/projected/c507592e-7a1a-4634-88a1-59ddf949c1dd-kube-api-access-sq2pr\") pod \"c507592e-7a1a-4634-88a1-59ddf949c1dd\" (UID: \"c507592e-7a1a-4634-88a1-59ddf949c1dd\") " Oct 31 03:30:29.673266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.672110 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c507592e-7a1a-4634-88a1-59ddf949c1dd-maas-api-service-ca" pod "c507592e-7a1a-4634-88a1-59ddf949c1dd" (UID: "c507592e-7a1a-4634-88a1-59ddf949c1dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:29.709903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.709802 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c507592e-7a1a-4634-88a1-59ddf949c1dd-kube-api-access-sq2pr" pod "c507592e-7a1a-4634-88a1-59ddf949c1dd" (UID: "c507592e-7a1a-4634-88a1-59ddf949c1dd"). InnerVolumeSpecName "kube-api-access-sq2pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:29.723579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.723284 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr"] Oct 31 03:30:29.770959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.770925 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c507592e-7a1a-4634-88a1-59ddf949c1dd-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.770959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.770960 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2pr\" (UniqueName: \"kubernetes.io/projected/c507592e-7a1a-4634-88a1-59ddf949c1dd-kube-api-access-sq2pr\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:29.787220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.784594 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" event={"ID":"c507592e-7a1a-4634-88a1-59ddf949c1dd","Type":"ContainerDied","Data":"e4c5986494d430815b1c71a354ac76e4783f56e568359d0e488bce835e0e316f"} Oct 31 03:30:29.787220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.784706 5520 scope.go:122] "RemoveContainer" containerID="29d62d098cdc170851d903c5c02e5411ecadf802c84c3e10e9c3d03e42ed1db7" Oct 31 03:30:29.787220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.784853 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g" Oct 31 03:30:29.829811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.826694 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr"] Oct 31 03:30:29.896247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.896170 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g"] Oct 31 03:30:29.896247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.896243 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890290-4r95g"] Oct 31 03:30:29.954742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.948240 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" path="/var/lib/kubelet/pods/c507592e-7a1a-4634-88a1-59ddf949c1dd/volumes" Oct 31 03:30:29.997408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.995588 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:29.997408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:29.997169 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:30.012594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.002560 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:30:30.043241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.034664 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:30.065247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.056295 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:30.249996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.249632 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:30:30.281112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.270581 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:30.324291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.323302 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:30.324291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.323373 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:30.348787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.348751 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f739d2c-54df-4030-90b6-9d30a03e62d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f739d2c-54df-4030-90b6-9d30a03e62d1-maas-api-service-ca\") pod \"9f739d2c-54df-4030-90b6-9d30a03e62d1\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " Oct 31 03:30:30.348922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.348816 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f739d2c-54df-4030-90b6-9d30a03e62d1-kube-api-access-wpv5r\" (UniqueName: \"kubernetes.io/projected/9f739d2c-54df-4030-90b6-9d30a03e62d1-kube-api-access-wpv5r\") pod \"9f739d2c-54df-4030-90b6-9d30a03e62d1\" (UID: \"9f739d2c-54df-4030-90b6-9d30a03e62d1\") " Oct 31 03:30:30.349592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.349552 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:30.349757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.349579 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f739d2c-54df-4030-90b6-9d30a03e62d1-maas-api-service-ca" pod "9f739d2c-54df-4030-90b6-9d30a03e62d1" (UID: "9f739d2c-54df-4030-90b6-9d30a03e62d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:30.352350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.352321 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f739d2c-54df-4030-90b6-9d30a03e62d1-kube-api-access-wpv5r" pod "9f739d2c-54df-4030-90b6-9d30a03e62d1" (UID: "9f739d2c-54df-4030-90b6-9d30a03e62d1"). InnerVolumeSpecName "kube-api-access-wpv5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:30.388265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.379822 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:30.388265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.383168 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:30.388265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.385346 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:30:30.451254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.451196 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e81ff07-0ad2-454f-b6cb-ed7663288119-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81ff07-0ad2-454f-b6cb-ed7663288119-maas-api-service-ca\") pod \"8e81ff07-0ad2-454f-b6cb-ed7663288119\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " Oct 31 03:30:30.451417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.451303 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81ff07-0ad2-454f-b6cb-ed7663288119-kube-api-access-ql7cx\" (UniqueName: \"kubernetes.io/projected/8e81ff07-0ad2-454f-b6cb-ed7663288119-kube-api-access-ql7cx\") pod \"8e81ff07-0ad2-454f-b6cb-ed7663288119\" (UID: \"8e81ff07-0ad2-454f-b6cb-ed7663288119\") " Oct 31 03:30:30.451616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.451574 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e81ff07-0ad2-454f-b6cb-ed7663288119-maas-api-service-ca" pod "8e81ff07-0ad2-454f-b6cb-ed7663288119" (UID: "8e81ff07-0ad2-454f-b6cb-ed7663288119"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:30.452256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.452238 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f739d2c-54df-4030-90b6-9d30a03e62d1-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:30.452344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.452261 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpv5r\" (UniqueName: \"kubernetes.io/projected/9f739d2c-54df-4030-90b6-9d30a03e62d1-kube-api-access-wpv5r\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:30.452344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.452280 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81ff07-0ad2-454f-b6cb-ed7663288119-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:30.453726 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.453706 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81ff07-0ad2-454f-b6cb-ed7663288119-kube-api-access-ql7cx" pod "8e81ff07-0ad2-454f-b6cb-ed7663288119" (UID: "8e81ff07-0ad2-454f-b6cb-ed7663288119"). InnerVolumeSpecName "kube-api-access-ql7cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:30.554316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.554219 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7cx\" (UniqueName: \"kubernetes.io/projected/8e81ff07-0ad2-454f-b6cb-ed7663288119-kube-api-access-ql7cx\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:30.795156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.795113 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" Oct 31 03:30:30.795657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.795106 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr" event={"ID":"8e81ff07-0ad2-454f-b6cb-ed7663288119","Type":"ContainerDied","Data":"1c3a07163b2acfcc2841b77dd47a51ce60fb6eaa7b83cc2f319246374adc624a"} Oct 31 03:30:30.795657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.795310 5520 scope.go:122] "RemoveContainer" containerID="cac61d88cb302808bbc63c00e1f3c3e235353e1d6a06a4f868b571ec23f8dc69" Oct 31 03:30:30.799657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.799615 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" event={"ID":"9f739d2c-54df-4030-90b6-9d30a03e62d1","Type":"ContainerDied","Data":"d05c13920b73f78bc2d952656e3c710fb937921bd6a77168c51ac2d984ee63d8"} Oct 31 03:30:30.799813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.799714 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr" Oct 31 03:30:30.812081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.810477 5520 scope.go:122] "RemoveContainer" containerID="c8a7037acb66d1009b6554ef0d6ee6c3d5ecc55d1b8d885afbad98ebb77347de" Oct 31 03:30:30.822911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.822892 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr"] Oct 31 03:30:30.842757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.842717 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-sqbfr"] Oct 31 03:30:30.850900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.850869 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr"] Oct 31 03:30:30.855613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:30.855586 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890290-zwfqr"] Oct 31 03:30:31.932573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:31.932535 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" path="/var/lib/kubelet/pods/8e81ff07-0ad2-454f-b6cb-ed7663288119/volumes" Oct 31 03:30:31.933185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:31.932983 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" path="/var/lib/kubelet/pods/9f739d2c-54df-4030-90b6-9d30a03e62d1/volumes" Oct 31 03:30:33.737254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.735287 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.765235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.763046 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:30:33.765235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.764180 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.868258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.868215 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.908228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.902693 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.933230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.909299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.991285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.970336 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:33.995225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.991750 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:30:33.995225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:33.992405 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.086229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.085797 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:34.125229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.117512 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.610229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.609670 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:30:34.855942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.801784 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:30:34.855942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.802183 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:34.855942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.802536 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:30:34.855942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.802596 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:30:34.855942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.803298 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:30:34.855942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:34.803812 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:30:35.009811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.004348 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.009811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.004799 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:30:35.009811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.006687 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Oct 31 03:30:35.009811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.007483 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.071914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.071880 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:30:35.072233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.072196 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:30:35.075971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.075948 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:35.228231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.221033 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:35.272451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.272360 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.273740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.273716 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:30:35.275377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.275323 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:35.276145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.276114 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:30:35.277358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.277333 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:30:35.277839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.277732 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.278010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.277985 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 31 03:30:35.278134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.278117 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:30:35.280534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.280510 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:35.282333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.282312 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:35.381516 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.381468 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:35.721835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.490042 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.722136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.624883 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.722990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.678696 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.727755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.727729 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:30:35.748238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.721279 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 31 03:30:35.815228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.813123 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.815228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.813543 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.815228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:35.814017 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.017282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.016048 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.229235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.228811 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:30:36.284289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.283310 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.297266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.292287 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:36.297266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.293260 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.376374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.376315 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.829227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.778242 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.830314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.799172 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:30:36.858388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.804182 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 31 03:30:36.858692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.816308 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.859387 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.816355 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:30:36.859794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.816376 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:30:36.967316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:36.967256 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.044248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.030177 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:30:37.045256 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:37.045223 5520 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.103s" Oct 31 03:30:37.045564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.045549 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:30:37.065256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.045733 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:30:37.239527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.239483 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:30:37.284326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.283676 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.305243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.304305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.305243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.304704 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:37.378622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.378584 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:37.408888 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.408860 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:37.849034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.848992 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.861880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.861842 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.866609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.862116 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.867949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.862141 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:37.875661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.862168 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.923346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.923301 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.976364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:37.976322 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.041473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.041425 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.098998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.098968 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:30:38.249230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.243926 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.279356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.279304 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.280019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.279996 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:30:38.318231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.284276 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.407228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.318527 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.407228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.318696 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.407228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.387325 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:38.846578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.846454 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.847878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.847569 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.854316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.853298 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.874239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.870466 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:30:38.874239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.874141 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:30:38.874971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.874941 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:30:38.882811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.882779 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.978349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:38.978293 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:39.046363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.046315 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:39.248354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.248306 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:30:39.249575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.249547 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:39.281174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.281022 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:39.438324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.415940 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:39.566217 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.550729 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:30:39.566217 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.553586 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:30:39.606539 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.602049 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:39.697554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.697521 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:30:39.743269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.737301 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 31 03:30:39.937003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.936955 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.040248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.938964 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:30:40.080358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.939722 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.080358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.941543 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.080358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.060038 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.080358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:39.981472 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.080358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.079286 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.257726 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.257623 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.376230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.375684 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.406469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.385780 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:30:40.481933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.481883 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.513350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.513260 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:30:40.513813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.513798 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:30:40.543960 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.543795 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:30:40.548373 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.548343 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:30:40.549999 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:30:40.549965 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:30:40.556025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.555114 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.601082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.601055 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:30:40.663298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.663263 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:30:40.672217 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.671506 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:30:40.697833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.697803 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:30:40.702963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.702939 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:30:40.707276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:40.707252 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:30:44.543763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.543732 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw"] Oct 31 03:30:44.545482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.545459 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:30:44.545637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.545622 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:30:44.545749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.545738 5520 cpu_manager.go:414] "containerMap: removing container" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.545833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.545822 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.545908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.545899 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:30:44.546001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.545990 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:30:44.546084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546075 5520 cpu_manager.go:414] "containerMap: removing container" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.546184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546168 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.546327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546314 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.546440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546425 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.546525 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546515 5520 cpu_manager.go:414] "containerMap: removing container" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.546595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546586 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.546690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546681 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.546762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546753 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.546835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546826 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.546908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.546899 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.547147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.547125 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.547267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.547256 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:30:44.547364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.547354 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:30:44.547445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.547437 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.547517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.547509 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:30:44.547588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.547579 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.547661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.547653 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81ff07-0ad2-454f-b6cb-ed7663288119" containerName="cleanup" Oct 31 03:30:44.564944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.564912 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw"] Oct 31 03:30:44.565127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.564978 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.575217 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.575168 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:30:44.575497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.575460 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:30:44.636034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.636001 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.636243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.636196 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.636298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.636275 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4s7\" (UniqueName: \"kubernetes.io/projected/1950d237-6669-414e-a9d6-e16c4c1a75e3-kube-api-access-dl4s7\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.737566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.737531 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.737737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.737659 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.737737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.737692 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4s7\" (UniqueName: \"kubernetes.io/projected/1950d237-6669-414e-a9d6-e16c4c1a75e3-kube-api-access-dl4s7\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.738318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.738294 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.740284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.740261 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.745747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.745723 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4s7\" (UniqueName: \"kubernetes.io/projected/1950d237-6669-414e-a9d6-e16c4c1a75e3-kube-api-access-dl4s7\") pod \"maas-api-revoke-sub-6096acc7-vkqzw\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:44.889851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:44.889761 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:45.246246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:45.245400 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw"] Oct 31 03:30:45.308570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:45.308535 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" event={"ID":"1950d237-6669-414e-a9d6-e16c4c1a75e3","Type":"ContainerStarted","Data":"6361078f2894948bb785c4d7e4dc6ed65b9c8f0d44c2767b55431289594a2cc9"} Oct 31 03:30:46.318103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:46.318065 5520 generic.go:356] "Generic (PLEG): container finished" podID="1950d237-6669-414e-a9d6-e16c4c1a75e3" containerID="bca9ccc2d11c2315bb17bb65b9dfa4e099ae5f51c0621cbe17d07e0351ed96fc" exitCode=0 Oct 31 03:30:46.318518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:46.318244 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" event={"ID":"1950d237-6669-414e-a9d6-e16c4c1a75e3","Type":"ContainerDied","Data":"bca9ccc2d11c2315bb17bb65b9dfa4e099ae5f51c0621cbe17d07e0351ed96fc"} Oct 31 03:30:47.622674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.622643 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:30:47.736686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.736646 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-xks-ca\") pod \"1950d237-6669-414e-a9d6-e16c4c1a75e3\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " Oct 31 03:30:47.736887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.736714 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-openshift-ca\") pod \"1950d237-6669-414e-a9d6-e16c4c1a75e3\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " Oct 31 03:30:47.736887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.736807 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1950d237-6669-414e-a9d6-e16c4c1a75e3-kube-api-access-dl4s7\" (UniqueName: \"kubernetes.io/projected/1950d237-6669-414e-a9d6-e16c4c1a75e3-kube-api-access-dl4s7\") pod \"1950d237-6669-414e-a9d6-e16c4c1a75e3\" (UID: \"1950d237-6669-414e-a9d6-e16c4c1a75e3\") " Oct 31 03:30:47.737090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.737066 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-openshift-ca" pod "1950d237-6669-414e-a9d6-e16c4c1a75e3" (UID: "1950d237-6669-414e-a9d6-e16c4c1a75e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:30:47.737596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.737577 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:47.739054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.739032 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-xks-ca" pod "1950d237-6669-414e-a9d6-e16c4c1a75e3" (UID: "1950d237-6669-414e-a9d6-e16c4c1a75e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:30:47.739360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.739336 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1950d237-6669-414e-a9d6-e16c4c1a75e3-kube-api-access-dl4s7" pod "1950d237-6669-414e-a9d6-e16c4c1a75e3" (UID: "1950d237-6669-414e-a9d6-e16c4c1a75e3"). InnerVolumeSpecName "kube-api-access-dl4s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:30:47.839659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.839573 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1950d237-6669-414e-a9d6-e16c4c1a75e3-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:47.839659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:47.839610 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4s7\" (UniqueName: \"kubernetes.io/projected/1950d237-6669-414e-a9d6-e16c4c1a75e3-kube-api-access-dl4s7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:30:48.335231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:48.335067 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" event={"ID":"1950d237-6669-414e-a9d6-e16c4c1a75e3","Type":"ContainerDied","Data":"6361078f2894948bb785c4d7e4dc6ed65b9c8f0d44c2767b55431289594a2cc9"} Oct 31 03:30:48.335231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:48.335168 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6361078f2894948bb785c4d7e4dc6ed65b9c8f0d44c2767b55431289594a2cc9" Oct 31 03:30:48.335475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:30:48.335281 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw" Oct 31 03:31:01.034230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.033418 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.064229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.063331 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.291285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.290453 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.303226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.296288 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.526255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.522344 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.635287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.634791 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.635477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.635291 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.710229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.709315 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.820229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.819989 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.821235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.820988 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:31:01.882227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.864292 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.912288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.887351 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:31:01.988229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:01.987415 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.068301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.067304 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.068301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.067892 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:31:02.359279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.296650 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.359279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.301508 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:31:02.359279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.304294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.359279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.315369 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:31:02.359279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.319339 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.359279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.320018 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:31:02.386229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.382399 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:31:02.554082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.531478 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.603261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.579548 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:31:02.636389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.636293 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:31:02.646379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.645488 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:31:02.715248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.710266 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.717296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.715701 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:31:02.791861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.791831 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:31:02.823323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.823287 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:31:02.824497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.823861 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:31:02.895280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.893745 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:31:02.911716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:02.905555 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:31:03.078226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.066544 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:31:03.078226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.067547 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.309835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.309763 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.325362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.325290 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.410324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.410271 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:31:03.411062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.411040 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:31:03.610285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.588344 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:03.610285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.588425 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.610285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.560636 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.650323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.617337 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:03.764258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.748706 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.832232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.830507 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:31:03.869386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.859510 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.869386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.862109 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.954230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:03.904735 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.102827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.073854 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:31:04.200803 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.122475 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.337950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.322178 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.613866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.354283 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.613866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.431289 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.613866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.436192 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.626701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.626499 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.627932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.627903 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.690253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.690222 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:04.690510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.690487 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.722053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.722001 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:31:04.722344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.722330 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:31:04.736810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.736782 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:31:04.740806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.740114 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:31:04.770944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.770344 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:04.804853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.804823 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:31:04.821362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.821334 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:31:04.859775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.859731 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.861281 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7"] Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.862149 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.862168 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.862261 5520 cpu_manager.go:414] "containerMap: removing container" podUID="1950d237-6669-414e-a9d6-e16c4c1a75e3" containerName="revoke-keys" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.862270 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1950d237-6669-414e-a9d6-e16c4c1a75e3" containerName="revoke-keys" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.862402 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f739d2c-54df-4030-90b6-9d30a03e62d1" containerName="cleanup" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.862421 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c507592e-7a1a-4634-88a1-59ddf949c1dd" containerName="cleanup" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.862432 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="1950d237-6669-414e-a9d6-e16c4c1a75e3" containerName="revoke-keys" Oct 31 03:31:04.864842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.863930 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.934230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.929422 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7"] Oct 31 03:31:04.934230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.929554 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:31:04.934230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.929601 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:31:04.934230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.929633 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:31:04.934230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.929718 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:04.989268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.985425 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk"] Oct 31 03:31:04.990968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.990946 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:04.992904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.992873 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:31:04.997588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:04.997570 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk"] Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001249 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001315 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001369 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001416 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001446 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvtz\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-kube-api-access-rpvtz\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001499 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001523 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001558 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001592 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001734 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001820 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001871 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001918 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.001960 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.002000 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72581b3c-56db-4876-b4e2-2feddb1126c0-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.002071 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8mq\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-kube-api-access-9r8mq\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.002119 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.002333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.002182 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.041552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.041528 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:31:05.042008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.041982 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:31:05.104017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.103952 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104025 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104049 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104074 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104103 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104126 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72581b3c-56db-4876-b4e2-2feddb1126c0-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104144 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8mq\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-kube-api-access-9r8mq\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104252 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104298 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104339 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104435 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104554 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104593 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104653 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvtz\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-kube-api-access-rpvtz\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104714 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.105151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104740 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.107340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.104801 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.107574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.107538 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.109908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.108082 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72581b3c-56db-4876-b4e2-2feddb1126c0-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.109908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.108870 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.109908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.109665 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.110151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.110037 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.111896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.111129 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.112371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.112107 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.112767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.112491 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.112767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.112695 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.112935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.112853 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.113428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.113121 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.115757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.114406 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.115928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.115504 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.122539 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.122506 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.126284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.125991 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvtz\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-kube-api-access-rpvtz\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.142441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.142384 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.148897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.148867 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.159657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.156287 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8mq\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-kube-api-access-9r8mq\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.164591 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.164559 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.257607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.257566 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:05.352719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.344783 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:05.573641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.573610 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7"] Oct 31 03:31:05.576472 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:31:05.576444 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372fbb32_2eb1_4f98_b790_2a93d25d08e9.slice/crio-b9476bcec9d85be6240deade3c1fbbe96c5ce191a50389a6810ca4f137a0ef1a WatchSource:0}: Error finding container b9476bcec9d85be6240deade3c1fbbe96c5ce191a50389a6810ca4f137a0ef1a: Status 404 returned error can't find the container with id b9476bcec9d85be6240deade3c1fbbe96c5ce191a50389a6810ca4f137a0ef1a Oct 31 03:31:05.579672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.579642 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:05.579755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.579725 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:05.579797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.579755 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:05.711878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.711820 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk"] Oct 31 03:31:05.713358 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:31:05.713324 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72581b3c_56db_4876_b4e2_2feddb1126c0.slice/crio-14fd71052f6cb0cfc7db0781d043fbf5e5c08e3db16679d6382a53f740d530b1 WatchSource:0}: Error finding container 14fd71052f6cb0cfc7db0781d043fbf5e5c08e3db16679d6382a53f740d530b1: Status 404 returned error can't find the container with id 14fd71052f6cb0cfc7db0781d043fbf5e5c08e3db16679d6382a53f740d530b1 Oct 31 03:31:05.719137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.719106 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:05.719361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.719336 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:05.719454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:05.719391 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:06.443008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.442975 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" event={"ID":"72581b3c-56db-4876-b4e2-2feddb1126c0","Type":"ContainerStarted","Data":"918a85cdb5fc16a63b8f354a83e0085ef3f417d66e443252d1b41c46171882a7"} Oct 31 03:31:06.443498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.443480 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" event={"ID":"72581b3c-56db-4876-b4e2-2feddb1126c0","Type":"ContainerStarted","Data":"14fd71052f6cb0cfc7db0781d043fbf5e5c08e3db16679d6382a53f740d530b1"} Oct 31 03:31:06.446059 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.446038 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" event={"ID":"372fbb32-2eb1-4f98-b790-2a93d25d08e9","Type":"ContainerStarted","Data":"517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115"} Oct 31 03:31:06.446219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.446192 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" event={"ID":"372fbb32-2eb1-4f98-b790-2a93d25d08e9","Type":"ContainerStarted","Data":"b9476bcec9d85be6240deade3c1fbbe96c5ce191a50389a6810ca4f137a0ef1a"} Oct 31 03:31:06.474226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.467192 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podStartSLOduration=2.467170908 podStartE2EDuration="2.467170908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.463407641 +0000 UTC m=+890.657930847" watchObservedRunningTime="2026-10-31 03:31:06.467170908 +0000 UTC m=+890.661694029" Oct 31 03:31:06.486019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.485945 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podStartSLOduration=2.48592441 podStartE2EDuration="2.48592441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.48392766 +0000 UTC m=+890.678450778" watchObservedRunningTime="2026-10-31 03:31:06.48592441 +0000 UTC m=+890.680447532" Oct 31 03:31:06.832287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.816109 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr"] Oct 31 03:31:06.837365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.835781 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr"] Oct 31 03:31:06.837365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.835911 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:06.860535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.860503 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:31:06.860938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.860899 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:31:06.946479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.946446 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxm2\" (UniqueName: \"kubernetes.io/projected/3a0bb301-2675-41e8-ae70-d69ad5709b0d-kube-api-access-hfxm2\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:06.946786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.946768 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:06.946922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:06.946908 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.048874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.048840 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.049446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.049425 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxm2\" (UniqueName: \"kubernetes.io/projected/3a0bb301-2675-41e8-ae70-d69ad5709b0d-kube-api-access-hfxm2\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.049604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.049430 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.050171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.050140 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.052886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.052863 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.058377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.058345 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxm2\" (UniqueName: \"kubernetes.io/projected/3a0bb301-2675-41e8-ae70-d69ad5709b0d-kube-api-access-hfxm2\") pod \"maas-api-revoke-sub-31167051-hlwfr\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.247690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.247652 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:07.258751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.258724 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:07.260499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.260458 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:31:07.260546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.260513 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 03:31:07.346292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.346260 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:07.347645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.347617 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:31:07.347750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.347677 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 03:31:07.727229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:07.727121 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr"] Oct 31 03:31:07.838401 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:31:07.838355 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0bb301_2675_41e8_ae70_d69ad5709b0d.slice/crio-40872db50ac7e32d354f24454a38b8b5bc57147d8b2682f83dc4fb60373262a0 WatchSource:0}: Error finding container 40872db50ac7e32d354f24454a38b8b5bc57147d8b2682f83dc4fb60373262a0: Status 404 returned error can't find the container with id 40872db50ac7e32d354f24454a38b8b5bc57147d8b2682f83dc4fb60373262a0 Oct 31 03:31:08.259175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:08.259138 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:31:08.259400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:08.259216 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 03:31:08.347025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:08.346723 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:31:08.347025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:08.346806 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 03:31:08.517319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:08.512273 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" event={"ID":"3a0bb301-2675-41e8-ae70-d69ad5709b0d","Type":"ContainerStarted","Data":"781ba11ef52787d5b456acb6eed2490e624ecccb904f04a3f7ee40a901f29457"} Oct 31 03:31:08.517319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:08.512361 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" event={"ID":"3a0bb301-2675-41e8-ae70-d69ad5709b0d","Type":"ContainerStarted","Data":"40872db50ac7e32d354f24454a38b8b5bc57147d8b2682f83dc4fb60373262a0"} Oct 31 03:31:08.572226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:08.565452 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" podStartSLOduration=2.565427854 podStartE2EDuration="2.565427854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:08.539705127 +0000 UTC m=+892.734228288" watchObservedRunningTime="2026-10-31 03:31:08.565427854 +0000 UTC m=+892.759950967" Oct 31 03:31:09.258753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:09.258713 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:31:09.259293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:09.258775 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 03:31:09.351522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:09.351385 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:31:09.352071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:09.351859 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 03:31:09.494230 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:31:09.485938 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:31:09.746353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:09.746320 5520 generic.go:356] "Generic (PLEG): container finished" podID="3a0bb301-2675-41e8-ae70-d69ad5709b0d" containerID="781ba11ef52787d5b456acb6eed2490e624ecccb904f04a3f7ee40a901f29457" exitCode=0 Oct 31 03:31:09.746559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:09.746509 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" event={"ID":"3a0bb301-2675-41e8-ae70-d69ad5709b0d","Type":"ContainerDied","Data":"781ba11ef52787d5b456acb6eed2490e624ecccb904f04a3f7ee40a901f29457"} Oct 31 03:31:10.258983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:10.258946 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:31:10.259458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:10.259007 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 03:31:10.346602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:10.346564 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:31:10.346778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:10.346627 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 03:31:11.201890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.201568 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:11.257891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.257856 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:31:11.258061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.257908 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 03:31:11.309230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.302859 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0bb301-2675-41e8-ae70-d69ad5709b0d-kube-api-access-hfxm2\" (UniqueName: \"kubernetes.io/projected/3a0bb301-2675-41e8-ae70-d69ad5709b0d-kube-api-access-hfxm2\") pod \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " Oct 31 03:31:11.309230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.303049 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-xks-ca\") pod \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " Oct 31 03:31:11.309230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.303084 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-openshift-ca\") pod \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\" (UID: \"3a0bb301-2675-41e8-ae70-d69ad5709b0d\") " Oct 31 03:31:11.341257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.338421 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-openshift-ca" pod "3a0bb301-2675-41e8-ae70-d69ad5709b0d" (UID: "3a0bb301-2675-41e8-ae70-d69ad5709b0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:31:11.351237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.345550 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:31:11.351237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.345618 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 03:31:11.351237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.348782 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0bb301-2675-41e8-ae70-d69ad5709b0d-kube-api-access-hfxm2" pod "3a0bb301-2675-41e8-ae70-d69ad5709b0d" (UID: "3a0bb301-2675-41e8-ae70-d69ad5709b0d"). InnerVolumeSpecName "kube-api-access-hfxm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:31:11.353362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.353324 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-xks-ca" pod "3a0bb301-2675-41e8-ae70-d69ad5709b0d" (UID: "3a0bb301-2675-41e8-ae70-d69ad5709b0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:31:11.507366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.431258 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfxm2\" (UniqueName: \"kubernetes.io/projected/3a0bb301-2675-41e8-ae70-d69ad5709b0d-kube-api-access-hfxm2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:11.507366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.431294 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:11.507366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.431310 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0bb301-2675-41e8-ae70-d69ad5709b0d-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:31:11.765020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.764931 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" Oct 31 03:31:11.765020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.764941 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr" event={"ID":"3a0bb301-2675-41e8-ae70-d69ad5709b0d","Type":"ContainerDied","Data":"40872db50ac7e32d354f24454a38b8b5bc57147d8b2682f83dc4fb60373262a0"} Oct 31 03:31:11.765020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:11.765020 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40872db50ac7e32d354f24454a38b8b5bc57147d8b2682f83dc4fb60373262a0" Oct 31 03:31:12.258378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:12.258343 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:31:12.258601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:12.258398 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 03:31:12.359418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:12.356827 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:31:12.359418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:12.356895 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 03:31:13.720237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:13.716653 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:13.720237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:13.718171 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:13.720237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:13.719458 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:31:13.791114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:13.791081 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:13.792816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:13.792655 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:13.797448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:13.797390 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:31:19.462229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.448161 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp"] Oct 31 03:31:19.462229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.449139 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3a0bb301-2675-41e8-ae70-d69ad5709b0d" containerName="revoke-keys" Oct 31 03:31:19.462229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.449160 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0bb301-2675-41e8-ae70-d69ad5709b0d" containerName="revoke-keys" Oct 31 03:31:19.462229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.449357 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0bb301-2675-41e8-ae70-d69ad5709b0d" containerName="revoke-keys" Oct 31 03:31:19.462229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.455311 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.691230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.680633 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.691230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.689724 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.691230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.689973 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.691230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.690050 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.691230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.690074 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.691230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.690141 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st22f\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-kube-api-access-st22f\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.718179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.718043 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.718487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.718448 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.718778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.718763 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.730229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.729816 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp"] Oct 31 03:31:19.821265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.819502 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:19.821265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.820170 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.851574 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.851715 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.851809 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.851894 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.851966 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.852021 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.852044 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.852097 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st22f\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-kube-api-access-st22f\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.852177 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.856225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.852715 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.977288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.908800 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.977288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.915314 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:19.977288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:19.916890 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:31:20.093230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.022726 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:20.178231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.131604 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:20.178231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.131837 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:20.178231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.132262 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.178231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.161626 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st22f\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-kube-api-access-st22f\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:20.187275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.182042 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:20.218230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.217863 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:20.307288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.304837 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:20.315828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.314353 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:31:20.413229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.412485 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:20.449342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.449295 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.624402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.621317 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:20.748636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.748352 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.825230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.820294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.825230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.820759 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:31:20.865247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.834802 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.865247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.835461 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:31:21.092248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.941839 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.092248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:20.942458 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.126425 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.138775 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.226290 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.231385 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.231462 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.233940 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.316636 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.318661 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.515259 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:31:21.709747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.517942 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:31:21.718232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.715735 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:21.741228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.736883 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:31:21.759929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.753006 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.759929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.759517 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:31:21.826282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.812197 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:31:21.826282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.821018 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.887243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.875312 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.887243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.876507 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.942451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:21.942390 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:31:22.169844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.134156 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.169844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.139561 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.201234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.200320 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:31:22.269865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.256046 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:22.327113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.278561 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:31:22.327113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.279305 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:31:22.327113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.317013 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.327113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.325151 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h"] Oct 31 03:31:22.761230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.733893 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.761230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.736464 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:31:22.856918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.856818 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.892647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.892603 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:31:22.944231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.903703 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.076103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.076048 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:23.076512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.076477 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:23.076711 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.076687 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:23.132314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.124938 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:31:23.132314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.132073 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5954fd-8678-46e5-aff6-1e044f097898-istio-podinfo\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.132314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.132180 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-istio-token\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.132314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.132269 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-socket\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.132664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.132444 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbl7b\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-kube-api-access-tbl7b\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.132664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.132563 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5954fd-8678-46e5-aff6-1e044f097898-istiod-ca-cert\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.167489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.167432 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-credential-socket\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.167698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.167550 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-certs\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.167969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.167944 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-data\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.168033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.168012 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-envoy\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.172231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.913845 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h"] Oct 31 03:31:23.172231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.169024 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:31:23.175116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.175073 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:31:23.235224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.914103 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:23.251062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:22.943328 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.744281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.290481 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.758230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.338641 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:31:23.759401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.340588 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.760070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.340971 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:23.771039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.414287 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.771853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.742345 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.829415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.758736 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.894374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.894316 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.920231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:23.895146 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:31:24.006629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.006516 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.160446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.160399 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:31:24.175337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.175288 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.203453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.203416 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1c45857e-openshift-default-dockercfg-x2l2n\"" Oct 31 03:31:24.322276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.321028 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbl7b\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-kube-api-access-tbl7b\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.459034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.194138 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbl7b\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-kube-api-access-tbl7b\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.459350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.459329 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5954fd-8678-46e5-aff6-1e044f097898-istiod-ca-cert\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.472782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.460359 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5954fd-8678-46e5-aff6-1e044f097898-istiod-ca-cert\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.473984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.473957 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-credential-socket\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.475609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.475578 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-certs\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.476440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.476040 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-certs\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.476628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.476086 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.477155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.476157 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-credential-socket\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.478161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.478140 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-data\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.502498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.486413 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-envoy\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.502498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.486563 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5954fd-8678-46e5-aff6-1e044f097898-istio-podinfo\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.502498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.486650 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-istio-token\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.502498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.486741 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-socket\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.502498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.487526 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-data\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.502498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.489349 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-socket\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.557749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.557618 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-envoy\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.656247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.588239 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" event={"ID":"ee92ee0d-060d-4e10-a543-7302c0e7e5e1","Type":"ContainerStarted","Data":"b49c3eaa1a430fb9e73e458bb8464bb13ab62cbce29f08fad91b026fafaa6f59"} Oct 31 03:31:24.656247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.588356 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:31:24.656247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.588399 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:31:24.656247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.588417 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp"] Oct 31 03:31:24.656247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.588477 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:31:24.701432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.689409 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:31:24.701432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.691787 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5954fd-8678-46e5-aff6-1e044f097898-istio-podinfo\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.753235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.743402 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:24.762226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.754092 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-istio-token\") pod \"e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:24.950392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.853517 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.980343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.879745 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:31:24.992229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.879775 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:31:25.061228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.879796 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:31:25.061228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:24.879817 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.061228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.042029 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:31:25.088229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.064916 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:25.154260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.145299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.196243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.183967 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.196243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.185849 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.196243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.193846 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 03:31:25.236801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.236698 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.321242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.321095 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:31:25.713235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.690334 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.714244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.714054 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.739261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.728004 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" event={"ID":"ee92ee0d-060d-4e10-a543-7302c0e7e5e1","Type":"ContainerStarted","Data":"7796aed021dfda83fa6f61fe8b0e0d1f4ebe218ef2a31cac33b4bb1bc1c44e7a"} Oct 31 03:31:25.905256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.902288 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.923595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.923533 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podStartSLOduration=6.923512607 podStartE2EDuration="6.923512607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:25.778793051 +0000 UTC m=+909.973316174" watchObservedRunningTime="2026-10-31 03:31:25.923512607 +0000 UTC m=+910.118035728" Oct 31 03:31:25.963411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.963325 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:31:26.030252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:25.984871 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.030826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.001965 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.053605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.053565 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:31:26.129674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.053926 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.134242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.053996 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:31:26.246857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.054017 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:31:26.374229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.205606 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.481287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.477451 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:26.481287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.477514 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:26.481287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.414948 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:26.481287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.219166 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.607450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.489315 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 31 03:31:26.798941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.798906 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:31:26.834284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.830641 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:31:26.917066 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.830957 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:26.948572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.948536 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:26.948872 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:26.948841 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.035581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.035483 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.064236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.058873 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.064236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.059736 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.065071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.065045 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:31:27.104230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.100902 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h"] Oct 31 03:31:27.165500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.165446 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.177235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.168451 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:31:27.237931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.237881 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:27.282329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.268839 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.283849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.283613 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:27.374746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.366276 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:31:27.420977 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.420925 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.466541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.466500 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:31:27.545652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.505029 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:27.545652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.505084 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:27.545652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.540102 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:31:27.568292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.561067 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:31:27.991229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.932582 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:28.043229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.932966 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:28.320159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:27.933010 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:28.320159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.037586 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:28.320159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.126524 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:28.323461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.126582 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:31:28.340604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.182409 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:31:28.340604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.182448 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:31:28.340604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.305370 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:31:28.548838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.546417 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:28.567234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.563884 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:28.619286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.594166 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:28.681189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.613630 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:31:28.705231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.647525 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:28.705920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.705872 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:28.869192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:28.869148 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:29.323583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.321099 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:31:29.324729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.324691 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:29.412388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.397711 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:29.504229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.414346 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:29.514653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.414386 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.414409 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.422149 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.538357 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:31:29.465511 5520 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.534s" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.538469 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.538515 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.538550 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.538578 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:31:29.551087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.538604 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr"] Oct 31 03:31:29.595433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.595336 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:29.595732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.595698 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:29.618011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.440049 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:29.698226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.668256 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Oct 31 03:31:29.738589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.738558 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:29.740480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.740450 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" event={"ID":"5e5954fd-8678-46e5-aff6-1e044f097898","Type":"ContainerStarted","Data":"be94683be297e6c9820331a4fc551e130bf29aa46b3be2e10bd998678e885154"} Oct 31 03:31:29.854756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.852333 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr"] Oct 31 03:31:29.995229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:29.960656 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgmk\" (UniqueName: \"kubernetes.io/projected/4616c3d2-5560-4f31-8309-07fbcf92f21b-kube-api-access-msgmk\") pod \"payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:30.052246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.043278 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4616c3d2-5560-4f31-8309-07fbcf92f21b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:30.217342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.210921 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:31:30.248018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.244237 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5aeec060-dockercfg-pzjql\"" Oct 31 03:31:30.248018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.244820 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5aeec060\"" Oct 31 03:31:30.274185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.268797 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4616c3d2-5560-4f31-8309-07fbcf92f21b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:30.274185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.269028 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgmk\" (UniqueName: \"kubernetes.io/projected/4616c3d2-5560-4f31-8309-07fbcf92f21b-kube-api-access-msgmk\") pod \"payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:30.387675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.357318 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4616c3d2-5560-4f31-8309-07fbcf92f21b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:30.387675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.361089 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:31:30.434224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.412676 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:30.434224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.414816 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:31:30.488286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.436368 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:30.488286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.436420 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:30.488286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.449176 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:31:30.488286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.457299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:30.488286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.461339 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:31:30.527897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.527858 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:31:30.575472 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.575431 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:30.611227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.584449 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:31:30.624951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.624809 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:30.630492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.630375 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgmk\" (UniqueName: \"kubernetes.io/projected/4616c3d2-5560-4f31-8309-07fbcf92f21b-kube-api-access-msgmk\") pod \"payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:30.635135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.635107 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" event={"ID":"5e5954fd-8678-46e5-aff6-1e044f097898","Type":"ContainerStarted","Data":"bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd"} Oct 31 03:31:30.662241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.662193 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:31:30.746940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.745951 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:31:30.752857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.752811 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:31:30.755615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.755580 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:31:30.838951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.838918 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:30.861275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:30.861190 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" podStartSLOduration=9.861172988 podStartE2EDuration="9.861172988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:30.858608885 +0000 UTC m=+915.053132024" watchObservedRunningTime="2026-10-31 03:31:30.861172988 +0000 UTC m=+915.055696110" Oct 31 03:31:31.085831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.085797 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:31.133228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.130589 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:31.574290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.566842 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:31.574290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.566943 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:31.655169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.654926 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr"] Oct 31 03:31:31.889300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.874321 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" event={"ID":"4616c3d2-5560-4f31-8309-07fbcf92f21b","Type":"ContainerStarted","Data":"92fd2fd39d54fa2e6ef5095d41bdce712bb8206f5777550c893d84972316d03a"} Oct 31 03:31:31.889300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.874758 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:31.957378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:31.957348 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:31:32.414864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:32.414774 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:32.414864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:32.414842 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:32.976229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:32.969992 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" event={"ID":"4616c3d2-5560-4f31-8309-07fbcf92f21b","Type":"ContainerStarted","Data":"c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5"} Oct 31 03:31:32.976229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:32.970111 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:33.018802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:33.018740 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" podStartSLOduration=4.018716275 podStartE2EDuration="4.018716275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:33.004241584 +0000 UTC m=+917.198764707" watchObservedRunningTime="2026-10-31 03:31:33.018716275 +0000 UTC m=+917.213239397" Oct 31 03:31:33.416875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:33.414672 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:33.416875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:33.414830 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:34.413971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.413934 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:34.414517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.414015 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:34.426664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.426633 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4"] Oct 31 03:31:34.431050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.431030 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:34.439776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.439742 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1c45857e\"" Oct 31 03:31:34.460258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.442173 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4"] Oct 31 03:31:34.560312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.560279 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg"] Oct 31 03:31:34.567138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.567108 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:34.599227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.594576 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1c45857e-dockercfg-hhprb\"" Oct 31 03:31:34.620236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.609358 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:34.620236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.609559 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfj4p\" (UniqueName: \"kubernetes.io/projected/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-kube-api-access-xfj4p\") pod \"payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:34.641232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.640297 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg"] Oct 31 03:31:34.727252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.715339 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:34.727252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.715536 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfj4p\" (UniqueName: \"kubernetes.io/projected/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-kube-api-access-xfj4p\") pod \"payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:34.727252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.715622 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:34.727252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.715811 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwld\" (UniqueName: \"kubernetes.io/projected/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-kube-api-access-wtwld\") pod \"payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:34.727252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.716989 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:34.823322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.820732 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:34.823322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.820889 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwld\" (UniqueName: \"kubernetes.io/projected/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-kube-api-access-wtwld\") pod \"payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:34.823322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.822304 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:34.888643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.870917 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfj4p\" (UniqueName: \"kubernetes.io/projected/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-kube-api-access-xfj4p\") pod \"payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:34.932239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.906500 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwld\" (UniqueName: \"kubernetes.io/projected/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-kube-api-access-wtwld\") pod \"payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:34.964597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:34.964324 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:35.066616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:35.066276 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:35.531283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:35.456325 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:35.531283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:35.457896 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:35.531283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:35.457938 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:35.610472 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:35.610291 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg"] Oct 31 03:31:35.663539 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:35.663439 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4"] Oct 31 03:31:35.663743 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:31:35.663695 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6fc2e2_87f0_40c8_ace6_6f44d7b2ee90.slice/crio-06e2af461c8e5cc8778e0dc54a6e16b67d53868343738722c22ea11e0999861b WatchSource:0}: Error finding container 06e2af461c8e5cc8778e0dc54a6e16b67d53868343738722c22ea11e0999861b: Status 404 returned error can't find the container with id 06e2af461c8e5cc8778e0dc54a6e16b67d53868343738722c22ea11e0999861b Oct 31 03:31:36.089229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.077897 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" event={"ID":"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90","Type":"ContainerStarted","Data":"dbf0074e7eea94747ed7b1410551bf004efece2282d2e2281098c144fb061a36"} Oct 31 03:31:36.092063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.089569 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" event={"ID":"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90","Type":"ContainerStarted","Data":"06e2af461c8e5cc8778e0dc54a6e16b67d53868343738722c22ea11e0999861b"} Oct 31 03:31:36.092063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.090950 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:36.104830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.104793 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" event={"ID":"95b931d4-91b2-49b8-947f-7ca9b7a5bbec","Type":"ContainerStarted","Data":"638e32ff956de1508576c58568b1827dcfc6518f53c9d06b233a1a328dd4cefe"} Oct 31 03:31:36.105120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.105096 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" event={"ID":"95b931d4-91b2-49b8-947f-7ca9b7a5bbec","Type":"ContainerStarted","Data":"b719d9eed20a9ce003ede82dd129830f473c7187db7deee7b31d37aa99bd9c4a"} Oct 31 03:31:36.106346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.106325 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:36.120361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.115988 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" podStartSLOduration=2.115969695 podStartE2EDuration="2.115969695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:36.112725836 +0000 UTC m=+920.307248958" watchObservedRunningTime="2026-10-31 03:31:36.115969695 +0000 UTC m=+920.310492855" Oct 31 03:31:36.213844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.213774 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" podStartSLOduration=2.213748127 podStartE2EDuration="2.213748127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:36.191228831 +0000 UTC m=+920.385751950" watchObservedRunningTime="2026-10-31 03:31:36.213748127 +0000 UTC m=+920.408271249" Oct 31 03:31:36.418219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.417974 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:36.418219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:36.418046 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:37.413868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:37.413830 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 31 03:31:37.414460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:37.413884 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 31 03:31:38.298886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.298844 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2"] Oct 31 03:31:38.304643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.304613 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.315845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.315821 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ed9442ec-serving-cert\"" Oct 31 03:31:38.327192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.327154 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2"] Oct 31 03:31:38.369149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.369113 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s66z8\" (UniqueName: \"kubernetes.io/projected/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-kube-api-access-s66z8\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.369332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.369317 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.369404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.369384 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.369507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.369492 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-metrics-certs\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.478235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.476557 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.480241 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s66z8\" (UniqueName: \"kubernetes.io/projected/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-kube-api-access-s66z8\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.480372 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.480455 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.480559 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-metrics-certs\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:31:38.481534 5520 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-serving-cert: secret "maas-api-e2e-migrate-ed9442ec-serving-cert" not found Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:31:38.481608 5520 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls podName:dd7e9ea8-3ce7-41f9-9faa-78bee5276f90 nodeName:}" failed. No retries permitted until 2026-10-31 03:31:38.981586541 +0000 UTC m=+923.176109640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls") pod "maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" (UID: "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90") : secret "maas-api-e2e-migrate-ed9442ec-serving-cert" not found Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.482470 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.505513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.494941 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-metrics-certs\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.513896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.513862 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s66z8\" (UniqueName: \"kubernetes.io/projected/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-kube-api-access-s66z8\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:38.535874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:38.535624 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:31:39.003147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:39.002481 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:39.035838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:39.035805 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls\") pod \"maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:39.225012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:39.224967 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:39.397993 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:31:39.397954 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:31:39.644757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:39.644735 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2"] Oct 31 03:31:39.645563 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:31:39.645537 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7e9ea8_3ce7_41f9_9faa_78bee5276f90.slice/crio-f5bdf4975f3745d4d1d0e8d62fda1a4e05d3e1caa07872ee0e826715b8ad8e26 WatchSource:0}: Error finding container f5bdf4975f3745d4d1d0e8d62fda1a4e05d3e1caa07872ee0e826715b8ad8e26: Status 404 returned error can't find the container with id f5bdf4975f3745d4d1d0e8d62fda1a4e05d3e1caa07872ee0e826715b8ad8e26 Oct 31 03:31:40.207154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:40.207114 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" event={"ID":"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90","Type":"ContainerStarted","Data":"f5bdf4975f3745d4d1d0e8d62fda1a4e05d3e1caa07872ee0e826715b8ad8e26"} Oct 31 03:31:41.258882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:41.258827 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" event={"ID":"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90","Type":"ContainerStarted","Data":"fd7bedce8ee5d9b450a9cd087e4826311981d1a15861aed079101c5069b5e832"} Oct 31 03:31:41.260849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:41.260823 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:41.289570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:41.289499 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" podStartSLOduration=2.704838193 podStartE2EDuration="3.289477571s" totalImagesPullingTime="584.639378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:41.287672724 +0000 UTC m=+925.482195848" watchObservedRunningTime="2026-10-31 03:31:41.289477571 +0000 UTC m=+925.484000693" Oct 31 03:31:45.055622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:45.055587 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:31:48.179781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:48.179745 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:31:48.180456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:48.180435 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:31:48.330682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:48.330655 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:31:54.673241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:54.669908 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:54.759225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:54.756382 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:54.836261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:54.825300 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:55.137413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.137356 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 03:31:55.138284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.138153 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.138544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.138260 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.181357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.181303 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.328357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.328301 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.346352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.346306 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.348966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.348479 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:31:55.350220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.349227 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.448227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.438134 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.681426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.681386 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:55.682728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.682704 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:31:55.740288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.708439 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 31 03:31:55.766961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.766326 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.793274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.788247 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:31:55.843245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.842677 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:55.843490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:55.843370 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:31:56.167300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.166048 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.251361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.189526 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:31:56.251361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.191530 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:31:56.251361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.202379 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.251361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.202635 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.364295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.357580 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:31:56.364295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.358010 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:31:56.364295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.358183 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:31:56.364295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.358738 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.460228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.459302 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.479370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.479323 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 31 03:31:56.637387 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.637353 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:31:56.637677 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.637662 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:31:56.637815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.637804 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:31:56.637936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.637925 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:31:56.638056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.638046 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:31:56.638169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.638160 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:31:56.780405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.780296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:31:56.916897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.846530 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:56.916897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:56.846863 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:57.193356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.193297 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.194015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.193980 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.194523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.194498 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:31:57.275285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.275251 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:31:57.275782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.275761 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:31:57.309239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.277827 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:31:57.397386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.397332 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.522368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.495116 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.580081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.580030 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.624278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.580437 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:31:57.629973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.580552 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:31:57.662605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.662558 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.888328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:57.888225 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.112356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.112311 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.195233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.195148 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.196457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.196429 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.226231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.207194 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.247753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.214645 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.248395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.236915 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.265357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.257324 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.331248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.322318 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:31:58.357239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.333283 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.447544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.436695 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:31:58.528433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.528403 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:31:58.621582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.620917 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:31:58.668540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.625379 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:31:58.934292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.913368 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:31:58.947237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.940540 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.947237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.945289 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.947237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:58.945378 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:31:59.208283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.152313 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:31:59.208283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.182298 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:59.208283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.182359 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.232422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.232355 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.273244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.252225 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.273244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.252262 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:31:59.273244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.252286 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:31:59.273244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.262516 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:31:59.273244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.272654 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:31:59.452230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.452082 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.678065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.677958 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.713691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.690820 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.741561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.741518 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.784554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.784501 5520 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:59.817228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.792280 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.865358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.865317 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:59.865629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.865604 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.921473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:31:59.921333 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.019551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.019503 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.021074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.021039 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.178546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.178495 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.293437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.293337 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:32:00.293998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.293337 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.294730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.293401 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.322691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.293433 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.323567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.295726 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:32:00.323965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.309385 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.386375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.354918 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.452416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.452353 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:32:00.513879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.513847 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:32:00.515461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.515439 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:32:00.516469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.516454 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:32:00.559550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.559466 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:32:00.715352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.715307 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.818279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:00.767344 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:32:01.023399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.023347 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.150293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.117008 5520 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:01.150293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.117084 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.192233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.179300 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.303232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.294917 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.328499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.328238 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.330587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.330563 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:32:01.369419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.369375 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr"] Oct 31 03:32:01.369643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.369379 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.439666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.439321 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:01.439666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.439388 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.447233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.446367 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.447233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.446887 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:32:01.563226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.562480 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:32:01.622229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.594740 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:01.879271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.872304 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.914714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.914681 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:32:01.950319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.950278 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.958986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:01.958917 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.038613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.038566 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.296316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.296276 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.455226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.305287 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:32:02.455226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.343707 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.455226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.397950 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:32:02.475246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.452478 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:32:02.606316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.554274 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:32:02.918281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.917303 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.918460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.918368 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.943229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.938067 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:32:02.968229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.953245 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.968229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:02.955582 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:32:03.038773 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.038734 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.148754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.148704 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.264225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.249514 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.456227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.456067 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:03.462850 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.462791 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.467379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.467344 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.468518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.468475 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.468997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.468961 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.471530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.471512 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:32:03.471748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.471734 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:32:03.558464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.558353 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.559281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.559256 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:32:03.636515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.636475 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:32:03.678311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.678268 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:32:03.874071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.873973 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:32:03.919082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.919042 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.967190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:03.965870 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:32:04.049669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.049615 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:04.067699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.067668 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:32:04.171231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.170412 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:04.171914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.171811 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:32:04.175975 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.173963 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:32:04.252236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.251536 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:04.252236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.252064 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:04.263661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.261834 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:32:04.286555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.286519 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:32:04.289757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.289710 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:32:04.309416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.302797 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:32:04.328181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.328153 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:04.342408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.342052 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 31 03:32:04.555871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.555842 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:32:04.559332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:04.558923 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:32:06.372227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.372154 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" podUID="4616c3d2-5560-4f31-8309-07fbcf92f21b" containerName="payload-pre-processing" containerID="cri-o://c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5" gracePeriod=25 Oct 31 03:32:06.701502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.701465 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:32:06.817692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.817653 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4616c3d2-5560-4f31-8309-07fbcf92f21b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4616c3d2-5560-4f31-8309-07fbcf92f21b-plugins-config-volume\") pod \"4616c3d2-5560-4f31-8309-07fbcf92f21b\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " Oct 31 03:32:06.817900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.817780 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4616c3d2-5560-4f31-8309-07fbcf92f21b-kube-api-access-msgmk\" (UniqueName: \"kubernetes.io/projected/4616c3d2-5560-4f31-8309-07fbcf92f21b-kube-api-access-msgmk\") pod \"4616c3d2-5560-4f31-8309-07fbcf92f21b\" (UID: \"4616c3d2-5560-4f31-8309-07fbcf92f21b\") " Oct 31 03:32:06.818104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.818077 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4616c3d2-5560-4f31-8309-07fbcf92f21b-plugins-config-volume" pod "4616c3d2-5560-4f31-8309-07fbcf92f21b" (UID: "4616c3d2-5560-4f31-8309-07fbcf92f21b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:06.818498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.818476 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4616c3d2-5560-4f31-8309-07fbcf92f21b-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:06.820667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.820621 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4616c3d2-5560-4f31-8309-07fbcf92f21b-kube-api-access-msgmk" pod "4616c3d2-5560-4f31-8309-07fbcf92f21b" (UID: "4616c3d2-5560-4f31-8309-07fbcf92f21b"). InnerVolumeSpecName "kube-api-access-msgmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:06.920019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.919924 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msgmk\" (UniqueName: \"kubernetes.io/projected/4616c3d2-5560-4f31-8309-07fbcf92f21b-kube-api-access-msgmk\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:06.945397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.943256 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf"] Oct 31 03:32:06.945397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.943995 5520 cpu_manager.go:414] "containerMap: removing container" podUID="4616c3d2-5560-4f31-8309-07fbcf92f21b" containerName="payload-pre-processing" Oct 31 03:32:06.945397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.944009 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4616c3d2-5560-4f31-8309-07fbcf92f21b" containerName="payload-pre-processing" Oct 31 03:32:06.945397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.944143 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="4616c3d2-5560-4f31-8309-07fbcf92f21b" containerName="payload-pre-processing" Oct 31 03:32:06.948772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.948748 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:06.956368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.955710 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-72b46d88-openshift-default-dockercfg-9x5j9\"" Oct 31 03:32:06.961149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:06.961126 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf"] Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028450 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d683eb31-9988-4255-b83b-de0ef5f032da-istiod-ca-cert\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028545 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-certs\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028572 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-data\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028635 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-istio-token\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028747 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-credential-socket\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028780 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-envoy\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028846 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-socket\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.028991 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d683eb31-9988-4255-b83b-de0ef5f032da-istio-podinfo\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.043232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.029103 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55qn8\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-kube-api-access-55qn8\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.137305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.132487 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55qn8\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-kube-api-access-55qn8\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.137305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.132563 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d683eb31-9988-4255-b83b-de0ef5f032da-istiod-ca-cert\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.137305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.132627 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-certs\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.137305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.133362 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-certs\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.137305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.134330 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d683eb31-9988-4255-b83b-de0ef5f032da-istiod-ca-cert\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.275481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.272506 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55qn8\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-kube-api-access-55qn8\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.381239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.372628 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-data\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.381239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.372738 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-istio-token\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.381239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.372930 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-credential-socket\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.381239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.372981 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-envoy\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.381239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.373055 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-socket\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.381239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.373222 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d683eb31-9988-4255-b83b-de0ef5f032da-istio-podinfo\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.381239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.380515 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-credential-socket\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.382892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.382862 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-data\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.384796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.384147 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-socket\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.386677 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.386640 5520 generic.go:356] "Generic (PLEG): container finished" podID="4616c3d2-5560-4f31-8309-07fbcf92f21b" containerID="c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5" exitCode=0 Oct 31 03:32:07.387509 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.387490 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" event={"ID":"4616c3d2-5560-4f31-8309-07fbcf92f21b","Type":"ContainerDied","Data":"c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5"} Oct 31 03:32:07.412626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.412586 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" event={"ID":"4616c3d2-5560-4f31-8309-07fbcf92f21b","Type":"ContainerDied","Data":"92fd2fd39d54fa2e6ef5095d41bdce712bb8206f5777550c893d84972316d03a"} Oct 31 03:32:07.412919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.412833 5520 scope.go:122] "RemoveContainer" containerID="c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5" Oct 31 03:32:07.413183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.407595 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr" Oct 31 03:32:07.459310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.459286 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d683eb31-9988-4255-b83b-de0ef5f032da-istio-podinfo\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.461757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.461734 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-envoy\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.463728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.463693 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-istio-token\") pod \"e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.487999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.487972 5520 scope.go:122] "RemoveContainer" containerID="c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5" Oct 31 03:32:07.488488 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:32:07.488461 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5\": container with ID starting with c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5 not found: ID does not exist" containerID="c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5" Oct 31 03:32:07.488618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.488498 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5"} err="failed to get container status \"c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5\": rpc error: code = NotFound desc = could not find container \"c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5\": container with ID starting with c328c7f4734cc4ecfc1cd3bd68245e4af82ed362d909522a8fd5455a799ec6d5 not found: ID does not exist" Oct 31 03:32:07.502935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.502894 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr"] Oct 31 03:32:07.508762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.508733 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5aeec060-64cf89dd6c-6fhrr"] Oct 31 03:32:07.568529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.568430 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:07.870167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.870142 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf"] Oct 31 03:32:07.872038 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:32:07.872005 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd683eb31_9988_4255_b83b_de0ef5f032da.slice/crio-9737ae0c583fd4bc3cbf7436375515ec9a7b52b684fe6b2a6c1b347b26cc3263 WatchSource:0}: Error finding container 9737ae0c583fd4bc3cbf7436375515ec9a7b52b684fe6b2a6c1b347b26cc3263: Status 404 returned error can't find the container with id 9737ae0c583fd4bc3cbf7436375515ec9a7b52b684fe6b2a6c1b347b26cc3263 Oct 31 03:32:07.875100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.875069 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:07.875237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.875140 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:07.875237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.875178 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:07.920998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:07.920962 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4616c3d2-5560-4f31-8309-07fbcf92f21b" path="/var/lib/kubelet/pods/4616c3d2-5560-4f31-8309-07fbcf92f21b/volumes" Oct 31 03:32:08.428240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:08.424725 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" event={"ID":"d683eb31-9988-4255-b83b-de0ef5f032da","Type":"ContainerStarted","Data":"07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651"} Oct 31 03:32:08.428240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:08.424793 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" event={"ID":"d683eb31-9988-4255-b83b-de0ef5f032da","Type":"ContainerStarted","Data":"9737ae0c583fd4bc3cbf7436375515ec9a7b52b684fe6b2a6c1b347b26cc3263"} Oct 31 03:32:08.467278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:08.452190 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" podStartSLOduration=2.452168783 podStartE2EDuration="2.452168783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:08.447803442 +0000 UTC m=+952.642326561" watchObservedRunningTime="2026-10-31 03:32:08.452168783 +0000 UTC m=+952.646691905" Oct 31 03:32:08.584227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:08.570269 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:08.596226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:08.593893 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:09.363160 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:32:09.363117 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:32:09.433377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:09.433348 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:09.434637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:09.434615 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:11.954009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:11.953978 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6"] Oct 31 03:32:11.958384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:11.958368 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:11.967782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:11.967747 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-72b46d88-dockercfg-84sm2\"" Oct 31 03:32:11.969771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:11.969702 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-72b46d88\"" Oct 31 03:32:11.982573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:11.975542 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6"] Oct 31 03:32:12.023812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.023772 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n"] Oct 31 03:32:12.028106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.028083 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.039541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.039515 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n"] Oct 31 03:32:12.091348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.091308 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6t69\" (UniqueName: \"kubernetes.io/projected/dce71e50-387a-464a-b741-1a3293043f66-kube-api-access-v6t69\") pod \"payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:12.091538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.091385 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ea74c2-7f27-4178-be29-708cb28bb9bd-plugins-config-volume\") pod \"payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.091621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.091604 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2cp\" (UniqueName: \"kubernetes.io/projected/a9ea74c2-7f27-4178-be29-708cb28bb9bd-kube-api-access-qr2cp\") pod \"payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.091992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.091966 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce71e50-387a-464a-b741-1a3293043f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:12.240277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.229425 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6t69\" (UniqueName: \"kubernetes.io/projected/dce71e50-387a-464a-b741-1a3293043f66-kube-api-access-v6t69\") pod \"payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:12.240277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.229512 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ea74c2-7f27-4178-be29-708cb28bb9bd-plugins-config-volume\") pod \"payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.240277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.229746 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2cp\" (UniqueName: \"kubernetes.io/projected/a9ea74c2-7f27-4178-be29-708cb28bb9bd-kube-api-access-qr2cp\") pod \"payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.240277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.229909 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce71e50-387a-464a-b741-1a3293043f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:12.240277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.230970 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce71e50-387a-464a-b741-1a3293043f66-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:12.266499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.266448 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ea74c2-7f27-4178-be29-708cb28bb9bd-plugins-config-volume\") pod \"payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.542282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.535919 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2cp\" (UniqueName: \"kubernetes.io/projected/a9ea74c2-7f27-4178-be29-708cb28bb9bd-kube-api-access-qr2cp\") pod \"payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.549237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.543589 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6t69\" (UniqueName: \"kubernetes.io/projected/dce71e50-387a-464a-b741-1a3293043f66-kube-api-access-v6t69\") pod \"payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:12.582647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.582181 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:12.651261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.651036 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:12.919492 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:32:12.919463 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce71e50_387a_464a_b741_1a3293043f66.slice/crio-2cf4d1e132257aec6a710d1d64b8cd61391a891682a45e082ff3c547fe605675 WatchSource:0}: Error finding container 2cf4d1e132257aec6a710d1d64b8cd61391a891682a45e082ff3c547fe605675: Status 404 returned error can't find the container with id 2cf4d1e132257aec6a710d1d64b8cd61391a891682a45e082ff3c547fe605675 Oct 31 03:32:12.919729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.919706 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6"] Oct 31 03:32:12.983129 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:32:12.983086 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ea74c2_7f27_4178_be29_708cb28bb9bd.slice/crio-33c8da82c9d77a81f20131fc07fc8ef4fbad826285bbe014dd0fc0af94b576a5 WatchSource:0}: Error finding container 33c8da82c9d77a81f20131fc07fc8ef4fbad826285bbe014dd0fc0af94b576a5: Status 404 returned error can't find the container with id 33c8da82c9d77a81f20131fc07fc8ef4fbad826285bbe014dd0fc0af94b576a5 Oct 31 03:32:12.987071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:12.987044 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n"] Oct 31 03:32:13.551179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.551119 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" event={"ID":"a9ea74c2-7f27-4178-be29-708cb28bb9bd","Type":"ContainerStarted","Data":"dfc809fcb226ac796ca846ce07ed84a0092fc282934973ff4e703bb6a2201d8f"} Oct 31 03:32:13.551414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.551249 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" event={"ID":"a9ea74c2-7f27-4178-be29-708cb28bb9bd","Type":"ContainerStarted","Data":"33c8da82c9d77a81f20131fc07fc8ef4fbad826285bbe014dd0fc0af94b576a5"} Oct 31 03:32:13.551414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.551378 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:13.553151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.553121 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" event={"ID":"dce71e50-387a-464a-b741-1a3293043f66","Type":"ContainerStarted","Data":"05ad93d0eacf77a2065fb0845583916accc873b5fee28e6c8aec8e4dfbd1a561"} Oct 31 03:32:13.553307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.553164 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" event={"ID":"dce71e50-387a-464a-b741-1a3293043f66","Type":"ContainerStarted","Data":"2cf4d1e132257aec6a710d1d64b8cd61391a891682a45e082ff3c547fe605675"} Oct 31 03:32:13.553307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.553258 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:13.572316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.572265 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" podStartSLOduration=2.57225038 podStartE2EDuration="2.57225038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:13.568524542 +0000 UTC m=+957.763047665" watchObservedRunningTime="2026-10-31 03:32:13.57225038 +0000 UTC m=+957.766773501" Oct 31 03:32:13.586100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:13.586054 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" podStartSLOduration=2.58603886 podStartE2EDuration="2.58603886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:13.583703459 +0000 UTC m=+957.778226616" watchObservedRunningTime="2026-10-31 03:32:13.58603886 +0000 UTC m=+957.780561980" Oct 31 03:32:18.009224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.007370 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d"] Oct 31 03:32:18.036886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.036858 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d"] Oct 31 03:32:18.037222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.037181 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.043000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.042959 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-72b46d88-serving-cert\"" Oct 31 03:32:18.157937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.157900 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-metrics-certs\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.158111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.157965 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-openshift-service-ca\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.158154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.158137 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-maas-api-tls\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.158266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.158231 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsb9n\" (UniqueName: \"kubernetes.io/projected/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-kube-api-access-hsb9n\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.264491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.260596 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-maas-api-tls\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.264491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.260690 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsb9n\" (UniqueName: \"kubernetes.io/projected/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-kube-api-access-hsb9n\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.264491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.261014 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-metrics-certs\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.264491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.261086 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-openshift-service-ca\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.264491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.262065 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-openshift-service-ca\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.285237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.275582 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-metrics-certs\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.310229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.303057 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-maas-api-tls\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.324074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.319881 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsb9n\" (UniqueName: \"kubernetes.io/projected/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-kube-api-access-hsb9n\") pod \"maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.361452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.361421 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:18.788353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:18.788318 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d"] Oct 31 03:32:19.700814 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:19.694295 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" event={"ID":"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c","Type":"ContainerStarted","Data":"56c042c1bf11d46ccc99560aaf610f986e7a4e32831596a3d5b5ca7a2415157d"} Oct 31 03:32:19.700814 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:19.694368 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" event={"ID":"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c","Type":"ContainerStarted","Data":"ddaf8d102264d3d5d1e639d258538d9087eb7c068fd4f014c3a366bd7af34889"} Oct 31 03:32:19.700814 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:19.695648 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:19.729651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:19.729590 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" podStartSLOduration=2.047469714 podStartE2EDuration="2.729572933s" totalImagesPullingTime="682.103219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:19.729529574 +0000 UTC m=+963.924052696" watchObservedRunningTime="2026-10-31 03:32:19.729572933 +0000 UTC m=+963.924096057" Oct 31 03:32:23.185898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:23.185831 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.267356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:23.267306 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.627290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:23.627143 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.010460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:23.973163 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:32:24.293302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.268984 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:24.354662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.330415 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.379315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.356399 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.411265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.380411 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:32:24.561247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.514290 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.622521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.622485 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:24.673244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.631292 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.673244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.632718 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:32:24.673244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.638640 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:32:24.865118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.865035 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:24.885345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.885296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.899766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:24.899721 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.062228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.062160 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.062734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.062695 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.220226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.198281 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:32:25.280578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.216588 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.280880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.216627 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:32:25.282380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.216654 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 31 03:32:25.282574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.269312 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.338528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.284909 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:32:25.339195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.339157 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:32:25.406265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.356574 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.406265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.357264 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:25.406265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.358330 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.406265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.358859 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 31 03:32:25.406265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.379591 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.406815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.406778 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.414223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.407346 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:32:25.586283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.585346 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.586283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.585813 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:32:25.679396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.679337 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.680156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.680115 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.680713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.680695 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:32:25.809897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.809867 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:32:25.872634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.872532 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg"] Oct 31 03:32:25.878330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.878286 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 31 03:32:25.911339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.895820 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.067765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:25.902529 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.076018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.063284 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:32:26.177806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.128228 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4"] Oct 31 03:32:26.199408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.199377 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:32:26.199688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.199674 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:32:26.222230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.199806 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:32:26.385608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.385583 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:32:26.487963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.326521 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:32:26.537547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.363882 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.538630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.538607 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:32:26.539361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.364570 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:32:26.539764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.384345 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.540109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.284366 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.572195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.551095 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:32:26.572195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.562350 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:32:26.572195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.487309 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.641555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.641526 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:26.642484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.642434 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.684372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.684281 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.694337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.694301 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:32:26.825882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.824329 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:32:26.892735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.891643 5520 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:26.892735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.891710 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.982956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:26.982912 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:27.076613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.076519 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:32:27.077106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.076641 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:27.078616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.076673 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:27.099313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.099285 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:32:27.375284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.333935 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:27.597230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.596596 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:32:27.726652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.726607 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:32:27.782757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.782717 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:27.783050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.783006 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:27.953239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:27.937434 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:32:28.020087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.019862 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:28.080309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.058945 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:32:28.389621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.122329 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:32:28.437264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.122639 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:32:28.469266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.122671 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.469266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.296993 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.604618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.604582 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:32:28.804227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.791624 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.850243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.849644 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.859486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.858412 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:32:28.938315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.938273 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:32:28.946377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.946341 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.947808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.947761 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:32:28.954023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:28.953984 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.022245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.022183 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:29.024361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.024331 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:32:29.189328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.186288 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded" Oct 31 03:32:29.404223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.384587 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:32:29.404223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.394279 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.434473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.434423 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.435751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.435720 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:32:29.455394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.455294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.611467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.611421 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.644954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.644924 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:32:29.789063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.788748 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.817259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.804633 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.994228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.953629 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:29.994228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.954172 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:32:29.994228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.984087 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.034230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.997750 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:32:30.034230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:29.998099 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.082297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.075181 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:30.082297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.075273 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.113054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.113004 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:32:30.419407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.419304 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:32:30.490452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.490402 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.512447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.466645 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.513003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.512970 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:32:30.602546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.602477 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.679054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.679014 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.809834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.809787 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:30.884276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.873912 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" podUID="95b931d4-91b2-49b8-947f-7ca9b7a5bbec" containerName="payload-processing" containerID="cri-o://638e32ff956de1508576c58568b1827dcfc6518f53c9d06b233a1a328dd4cefe" gracePeriod=25 Oct 31 03:32:30.977388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.977351 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:31.004750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.004727 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:32:31.005001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.004989 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:32:31.005252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.005239 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:32:31.005409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.005400 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:32:31.006013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:30.983509 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.043619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.003449 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.044521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.003556 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.044948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.001294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.057593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.057563 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:32:31.099252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.078351 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.149605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.147008 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.233987 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.231675 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" podUID="7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" containerName="payload-pre-processing" containerID="cri-o://dbf0074e7eea94747ed7b1410551bf004efece2282d2e2281098c144fb061a36" gracePeriod=25 Oct 31 03:32:31.533556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.485042 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.556391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.556334 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.660929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.604065 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.903248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.898824 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:31.903248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:31.898959 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.053863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.053820 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.068252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.061024 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.068252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.061256 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:32:32.068252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.061470 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.088733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.087295 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.131007 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.130908 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.151001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.147845 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.279234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.275368 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:32:32.279234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.275443 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:32:32.456916 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.456847 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.491304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.490686 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:32.865620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:32.865503 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:32:33.037946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.037906 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:32:33.165518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.061802 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:33.213228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.132417 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:33.213228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.211602 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:32:33.285366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.285324 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:32:33.355617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.331325 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:33.355617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.354727 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:32:33.396398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.396351 5520 generic.go:356] "Generic (PLEG): container finished" podID="7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" containerID="dbf0074e7eea94747ed7b1410551bf004efece2282d2e2281098c144fb061a36" exitCode=0 Oct 31 03:32:33.425295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.398654 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:32:33.425650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.425606 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" event={"ID":"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90","Type":"ContainerDied","Data":"dbf0074e7eea94747ed7b1410551bf004efece2282d2e2281098c144fb061a36"} Oct 31 03:32:33.505439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.505389 5520 generic.go:356] "Generic (PLEG): container finished" podID="95b931d4-91b2-49b8-947f-7ca9b7a5bbec" containerID="638e32ff956de1508576c58568b1827dcfc6518f53c9d06b233a1a328dd4cefe" exitCode=0 Oct 31 03:32:33.509943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.509911 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" event={"ID":"95b931d4-91b2-49b8-947f-7ca9b7a5bbec","Type":"ContainerDied","Data":"638e32ff956de1508576c58568b1827dcfc6518f53c9d06b233a1a328dd4cefe"} Oct 31 03:32:33.546630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.546590 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:32:33.552227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.549889 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:32:33.552227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.550227 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:32:33.552227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.551551 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:32:33.627228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.623144 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2"] Oct 31 03:32:33.627719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.627681 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" podUID="dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" containerName="maas-api" containerID="cri-o://fd7bedce8ee5d9b450a9cd087e4826311981d1a15861aed079101c5069b5e832" gracePeriod=30 Oct 31 03:32:33.785228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.736858 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:32:33.785228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.784312 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:32:33.822535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.807763 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:32:33.889085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.889043 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:33.933151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.931022 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-plugins-config-volume\") pod \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " Oct 31 03:32:33.933151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.931088 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-kube-api-access-xfj4p\" (UniqueName: \"kubernetes.io/projected/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-kube-api-access-xfj4p\") pod \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\" (UID: \"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90\") " Oct 31 03:32:33.962239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.941945 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-plugins-config-volume" pod "7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" (UID: "7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:33.991225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:33.988514 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-kube-api-access-xfj4p" pod "7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" (UID: "7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90"). InnerVolumeSpecName "kube-api-access-xfj4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:34.034942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.033490 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:34.034942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.033528 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfj4p\" (UniqueName: \"kubernetes.io/projected/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90-kube-api-access-xfj4p\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:34.375914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.350081 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:32:34.385590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.382633 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:34.389688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.389628 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:34.620650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.620609 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:34.621290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.621261 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:34.766698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.763587 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:34.766698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.763634 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl"] Oct 31 03:32:34.766698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.765215 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" containerName="payload-pre-processing" Oct 31 03:32:34.766698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.765235 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" containerName="payload-pre-processing" Oct 31 03:32:34.766698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.765792 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" containerName="payload-pre-processing" Oct 31 03:32:34.838215 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.837952 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:32:34.847910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.847531 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-kube-api-access-wtwld\" (UniqueName: \"kubernetes.io/projected/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-kube-api-access-wtwld\") pod \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " Oct 31 03:32:34.863048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.863004 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl"] Oct 31 03:32:34.863397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.863381 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:32:34.863648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.863624 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:32:34.863831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.863819 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:32:34.863989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.863974 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:32:34.864416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.864401 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:32:34.864570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.864558 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:32:34.864709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.864697 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:32:34.865399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.865358 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" event={"ID":"7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90","Type":"ContainerDied","Data":"06e2af461c8e5cc8778e0dc54a6e16b67d53868343738722c22ea11e0999861b"} Oct 31 03:32:34.865489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.865458 5520 scope.go:122] "RemoveContainer" containerID="dbf0074e7eea94747ed7b1410551bf004efece2282d2e2281098c144fb061a36" Oct 31 03:32:34.865976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.865958 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4" Oct 31 03:32:34.866076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.866062 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.866195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.863728 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-plugins-config-volume\") pod \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\" (UID: \"95b931d4-91b2-49b8-947f-7ca9b7a5bbec\") " Oct 31 03:32:34.877293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.864266 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-plugins-config-volume" pod "95b931d4-91b2-49b8-947f-7ca9b7a5bbec" (UID: "95b931d4-91b2-49b8-947f-7ca9b7a5bbec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:34.886613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.864813 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-kube-api-access-wtwld" pod "95b931d4-91b2-49b8-947f-7ca9b7a5bbec" (UID: "95b931d4-91b2-49b8-947f-7ca9b7a5bbec"). InnerVolumeSpecName "kube-api-access-wtwld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:34.893886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.893854 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" event={"ID":"95b931d4-91b2-49b8-947f-7ca9b7a5bbec","Type":"ContainerDied","Data":"b719d9eed20a9ce003ede82dd129830f473c7187db7deee7b31d37aa99bd9c4a"} Oct 31 03:32:34.894157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.894139 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg" Oct 31 03:32:34.942057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.942015 5520 generic.go:356] "Generic (PLEG): container finished" podID="dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" containerID="fd7bedce8ee5d9b450a9cd087e4826311981d1a15861aed079101c5069b5e832" exitCode=0 Oct 31 03:32:34.942948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.942089 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" event={"ID":"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90","Type":"ContainerDied","Data":"fd7bedce8ee5d9b450a9cd087e4826311981d1a15861aed079101c5069b5e832"} Oct 31 03:32:34.953977 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.953940 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h"] Oct 31 03:32:34.954182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.954156 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" podUID="5e5954fd-8678-46e5-aff6-1e044f097898" containerName="istio-proxy" containerID="cri-o://bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd" gracePeriod=30 Oct 31 03:32:34.960738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.959955 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.966543 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e192091-openshift-default-dockercfg-v9mmv\"" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.977358 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-data\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.977397 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25310c-5979-447d-9dcc-678bef6dbfae-istiod-ca-cert\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.977551 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-podinfo\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.977817 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-token\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.977894 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-envoy\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.977973 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-socket\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.978158 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-certs\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.978368 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgdt\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-kube-api-access-8hgdt\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.978400 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-credential-socket\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.978512 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:34.983650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.978529 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwld\" (UniqueName: \"kubernetes.io/projected/95b931d4-91b2-49b8-947f-7ca9b7a5bbec-kube-api-access-wtwld\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:34.991362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:34.991323 5520 scope.go:122] "RemoveContainer" containerID="638e32ff956de1508576c58568b1827dcfc6518f53c9d06b233a1a328dd4cefe" Oct 31 03:32:35.302987 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.302899 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-token\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.303339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.303317 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-envoy\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.303510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.303493 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-socket\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.303712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.303697 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-certs\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.303930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.303914 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgdt\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-kube-api-access-8hgdt\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.304078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.304061 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-credential-socket\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.304248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.304232 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-data\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.304370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.304355 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25310c-5979-447d-9dcc-678bef6dbfae-istiod-ca-cert\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.304547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.304532 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-podinfo\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.309530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.308363 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-podinfo\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.319485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.312049 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-envoy\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.319485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.312335 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-socket\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.319485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.312590 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-certs\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.320301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.313065 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-credential-socket\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.320301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.320062 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-data\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.322376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.320986 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25310c-5979-447d-9dcc-678bef6dbfae-istiod-ca-cert\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.345697 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.345621 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-token\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.352563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.352537 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:32:35.381350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.358803 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgdt\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-kube-api-access-8hgdt\") pod \"e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:35.430899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.430812 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4"] Oct 31 03:32:35.436448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.436420 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c45857e-78497fd579-lrvh4"] Oct 31 03:32:35.441934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.441891 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg"] Oct 31 03:32:35.446933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.446900 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c45857e-ccdbdb5f9-z54kg"] Oct 31 03:32:35.525915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.525881 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-kube-api-access-s66z8\" (UniqueName: \"kubernetes.io/projected/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-kube-api-access-s66z8\") pod \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " Oct 31 03:32:35.526254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.526235 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-metrics-certs\") pod \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " Oct 31 03:32:35.526539 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.526523 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls\") pod \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " Oct 31 03:32:35.526769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.526746 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-openshift-service-ca\") pod \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\" (UID: \"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90\") " Oct 31 03:32:35.532640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.532609 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-openshift-service-ca" pod "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" (UID: "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:35.536775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.536748 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-kube-api-access-s66z8" pod "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" (UID: "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90"). InnerVolumeSpecName "kube-api-access-s66z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:35.542422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.542381 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-metrics-certs" pod "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" (UID: "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:35.546440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.546378 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls" pod "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" (UID: "dd7e9ea8-3ce7-41f9-9faa-78bee5276f90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:35.638638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.638600 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:35.638638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.638640 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:35.638885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.638656 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:35.638885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.638673 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s66z8\" (UniqueName: \"kubernetes.io/projected/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90-kube-api-access-s66z8\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:35.661436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:35.661402 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:36.147513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.147482 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90" path="/var/lib/kubelet/pods/7e6fc2e2-87f0-40c8-ace6-6f44d7b2ee90/volumes" Oct 31 03:32:36.148184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.148086 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b931d4-91b2-49b8-947f-7ca9b7a5bbec" path="/var/lib/kubelet/pods/95b931d4-91b2-49b8-947f-7ca9b7a5bbec/volumes" Oct 31 03:32:36.148383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.148347 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:36.148573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.148554 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:36.148714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.148695 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:36.148864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.148770 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl"] Oct 31 03:32:36.155674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.155650 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" event={"ID":"5e25310c-5979-447d-9dcc-678bef6dbfae","Type":"ContainerStarted","Data":"80165d1cd40965d7c5a303cda99666221236999d5737c403dc0adbb947b46f2e"} Oct 31 03:32:36.159419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.159399 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" event={"ID":"dd7e9ea8-3ce7-41f9-9faa-78bee5276f90","Type":"ContainerDied","Data":"f5bdf4975f3745d4d1d0e8d62fda1a4e05d3e1caa07872ee0e826715b8ad8e26"} Oct 31 03:32:36.159546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.159536 5520 scope.go:122] "RemoveContainer" containerID="fd7bedce8ee5d9b450a9cd087e4826311981d1a15861aed079101c5069b5e832" Oct 31 03:32:36.159675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.159472 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2" Oct 31 03:32:36.182986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.182960 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2"] Oct 31 03:32:36.189799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:36.188664 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed9442ec-674bd84cbd-64kk2"] Oct 31 03:32:37.171939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:37.171901 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" event={"ID":"5e25310c-5979-447d-9dcc-678bef6dbfae","Type":"ContainerStarted","Data":"8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238"} Oct 31 03:32:37.196355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:37.196290 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" podStartSLOduration=6.196268516 podStartE2EDuration="6.196268516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:37.192767268 +0000 UTC m=+981.387290391" watchObservedRunningTime="2026-10-31 03:32:37.196268516 +0000 UTC m=+981.390791638" Oct 31 03:32:37.662770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:37.662728 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:37.670424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:37.670391 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:37.919087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:37.919005 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" path="/var/lib/kubelet/pods/dd7e9ea8-3ce7-41f9-9faa-78bee5276f90/volumes" Oct 31 03:32:38.183676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:38.183646 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:38.184935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:38.184901 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:32:39.365808 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:32:39.365769 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:32:40.095178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.095128 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c"] Oct 31 03:32:40.095894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.095872 5520 cpu_manager.go:414] "containerMap: removing container" podUID="95b931d4-91b2-49b8-947f-7ca9b7a5bbec" containerName="payload-processing" Oct 31 03:32:40.096011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.095896 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b931d4-91b2-49b8-947f-7ca9b7a5bbec" containerName="payload-processing" Oct 31 03:32:40.096011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.095940 5520 cpu_manager.go:414] "containerMap: removing container" podUID="dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" containerName="maas-api" Oct 31 03:32:40.096011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.095949 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" containerName="maas-api" Oct 31 03:32:40.096163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.096130 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b931d4-91b2-49b8-947f-7ca9b7a5bbec" containerName="payload-processing" Oct 31 03:32:40.096163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.096151 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7e9ea8-3ce7-41f9-9faa-78bee5276f90" containerName="maas-api" Oct 31 03:32:40.101040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.101016 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.106344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.106314 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e192091\"" Oct 31 03:32:40.108895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.108857 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e192091-dockercfg-59z7z\"" Oct 31 03:32:40.116396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.116363 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c"] Oct 31 03:32:40.167848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.167818 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv"] Oct 31 03:32:40.173734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.173709 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:40.224745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.221480 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv"] Oct 31 03:32:40.314226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.313260 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054e4608-7e1c-431f-a929-1c99c2d13456-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.314226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.313323 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j926r\" (UniqueName: \"kubernetes.io/projected/054e4608-7e1c-431f-a929-1c99c2d13456-kube-api-access-j926r\") pod \"payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.546239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.542660 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054e4608-7e1c-431f-a929-1c99c2d13456-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.546239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.542732 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j926r\" (UniqueName: \"kubernetes.io/projected/054e4608-7e1c-431f-a929-1c99c2d13456-kube-api-access-j926r\") pod \"payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.546239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.542821 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e192091-5946b58f57-57kcv\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:40.546239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.543162 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpsdl\" (UniqueName: \"kubernetes.io/projected/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-kube-api-access-kpsdl\") pod \"payload-processing-e2e-ait-0e192091-5946b58f57-57kcv\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:40.595610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.595497 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054e4608-7e1c-431f-a929-1c99c2d13456-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.618252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.610247 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j926r\" (UniqueName: \"kubernetes.io/projected/054e4608-7e1c-431f-a929-1c99c2d13456-kube-api-access-j926r\") pod \"payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.687235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.686609 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e192091-5946b58f57-57kcv\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:40.687235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.686823 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpsdl\" (UniqueName: \"kubernetes.io/projected/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-kube-api-access-kpsdl\") pod \"payload-processing-e2e-ait-0e192091-5946b58f57-57kcv\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:40.687673 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.687650 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e192091-5946b58f57-57kcv\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:40.695143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.695112 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpsdl\" (UniqueName: \"kubernetes.io/projected/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-kube-api-access-kpsdl\") pod \"payload-processing-e2e-ait-0e192091-5946b58f57-57kcv\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:40.733250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.733194 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:40.820241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:40.817117 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:41.135922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:41.133843 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c"] Oct 31 03:32:41.138324 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:32:41.138277 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054e4608_7e1c_431f_a929_1c99c2d13456.slice/crio-240e19c49c7306e6b37f49bffaac91d3958d2601d1804c3634c4c7a59951b8e7 WatchSource:0}: Error finding container 240e19c49c7306e6b37f49bffaac91d3958d2601d1804c3634c4c7a59951b8e7: Status 404 returned error can't find the container with id 240e19c49c7306e6b37f49bffaac91d3958d2601d1804c3634c4c7a59951b8e7 Oct 31 03:32:41.166813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:41.166777 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv"] Oct 31 03:32:41.185092 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:32:41.185052 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8e94bf_bb8e_43d0_b3e3_6f00dda2316a.slice/crio-918862be37851c3ba9cb6b9bfea8f184a0ab3588c7c8cc749a7e948a74a924d9 WatchSource:0}: Error finding container 918862be37851c3ba9cb6b9bfea8f184a0ab3588c7c8cc749a7e948a74a924d9: Status 404 returned error can't find the container with id 918862be37851c3ba9cb6b9bfea8f184a0ab3588c7c8cc749a7e948a74a924d9 Oct 31 03:32:41.231095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:41.231066 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" event={"ID":"054e4608-7e1c-431f-a929-1c99c2d13456","Type":"ContainerStarted","Data":"240e19c49c7306e6b37f49bffaac91d3958d2601d1804c3634c4c7a59951b8e7"} Oct 31 03:32:41.232678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:41.232651 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" event={"ID":"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a","Type":"ContainerStarted","Data":"918862be37851c3ba9cb6b9bfea8f184a0ab3588c7c8cc749a7e948a74a924d9"} Oct 31 03:32:41.977080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:41.972420 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.027682 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-credential-socket\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.027775 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-certs\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.027917 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5954fd-8678-46e5-aff6-1e044f097898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5954fd-8678-46e5-aff6-1e044f097898-istiod-ca-cert\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.027968 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-data\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.028089 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-kube-api-access-tbl7b\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-kube-api-access-tbl7b\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.028142 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e5954fd-8678-46e5-aff6-1e044f097898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5954fd-8678-46e5-aff6-1e044f097898-istio-podinfo\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.028188 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-envoy\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.028284 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-socket\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.028322 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-istio-token\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-istio-token\") pod \"5e5954fd-8678-46e5-aff6-1e044f097898\" (UID: \"5e5954fd-8678-46e5-aff6-1e044f097898\") " Oct 31 03:32:42.030835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.030663 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5954fd-8678-46e5-aff6-1e044f097898-istiod-ca-cert" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:42.030927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.030904 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-certs" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:42.031095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.031073 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-data" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:42.031791 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.031252 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-credential-socket" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:42.031791 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.031441 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-socket" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:42.079356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.079318 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-kube-api-access-tbl7b" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "kube-api-access-tbl7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:42.123501 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.123274 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-envoy" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:42.123501 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.123427 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e5954fd-8678-46e5-aff6-1e044f097898-istio-podinfo" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:32:42.125566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.125536 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-istio-token" pod "5e5954fd-8678-46e5-aff6-1e044f097898" (UID: "5e5954fd-8678-46e5-aff6-1e044f097898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132182 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbl7b\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-kube-api-access-tbl7b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132233 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5954fd-8678-46e5-aff6-1e044f097898-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132250 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132266 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132279 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5954fd-8678-46e5-aff6-1e044f097898-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132293 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132307 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132331 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5954fd-8678-46e5-aff6-1e044f097898-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.133255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.132345 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5954fd-8678-46e5-aff6-1e044f097898-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:42.242847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.242760 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" event={"ID":"054e4608-7e1c-431f-a929-1c99c2d13456","Type":"ContainerStarted","Data":"3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848"} Oct 31 03:32:42.243086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.243061 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:42.244317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.244293 5520 generic.go:356] "Generic (PLEG): container finished" podID="5e5954fd-8678-46e5-aff6-1e044f097898" containerID="bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd" exitCode=0 Oct 31 03:32:42.244451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.244339 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" Oct 31 03:32:42.244451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.244378 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" event={"ID":"5e5954fd-8678-46e5-aff6-1e044f097898","Type":"ContainerDied","Data":"bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd"} Oct 31 03:32:42.244451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.244429 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h" event={"ID":"5e5954fd-8678-46e5-aff6-1e044f097898","Type":"ContainerDied","Data":"be94683be297e6c9820331a4fc551e130bf29aa46b3be2e10bd998678e885154"} Oct 31 03:32:42.244608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.244458 5520 scope.go:122] "RemoveContainer" containerID="bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd" Oct 31 03:32:42.249374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.249347 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" event={"ID":"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a","Type":"ContainerStarted","Data":"de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791"} Oct 31 03:32:42.249492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.249465 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:42.266370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.266346 5520 scope.go:122] "RemoveContainer" containerID="bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd" Oct 31 03:32:42.266897 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:32:42.266869 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd\": container with ID starting with bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd not found: ID does not exist" containerID="bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd" Oct 31 03:32:42.267016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.266907 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd"} err="failed to get container status \"bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd\": rpc error: code = NotFound desc = could not find container \"bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd\": container with ID starting with bf966e3bb35786684d92f9f78439ad6ede7f2076a9b414d068ffd91ca135f6bd not found: ID does not exist" Oct 31 03:32:42.267016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.266949 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" podStartSLOduration=2.266933084 podStartE2EDuration="2.266933084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:42.266406635 +0000 UTC m=+986.460929758" watchObservedRunningTime="2026-10-31 03:32:42.266933084 +0000 UTC m=+986.461456207" Oct 31 03:32:42.287022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.286976 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" podStartSLOduration=2.286963654 podStartE2EDuration="2.286963654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:42.282009546 +0000 UTC m=+986.476532697" watchObservedRunningTime="2026-10-31 03:32:42.286963654 +0000 UTC m=+986.481486774" Oct 31 03:32:42.300233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.300177 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h"] Oct 31 03:32:42.303972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:42.303950 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1c45857e-openshift-default-fcdfbc4bf-ph92h"] Oct 31 03:32:43.919425 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:43.919392 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5954fd-8678-46e5-aff6-1e044f097898" path="/var/lib/kubelet/pods/5e5954fd-8678-46e5-aff6-1e044f097898/volumes" Oct 31 03:32:45.138892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.138857 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n"] Oct 31 03:32:45.153547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.153516 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6"] Oct 31 03:32:45.451480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.451435 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb"] Oct 31 03:32:45.462377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.462350 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5e5954fd-8678-46e5-aff6-1e044f097898" containerName="istio-proxy" Oct 31 03:32:45.462606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.462590 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5954fd-8678-46e5-aff6-1e044f097898" containerName="istio-proxy" Oct 31 03:32:45.462968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.462952 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5954fd-8678-46e5-aff6-1e044f097898" containerName="istio-proxy" Oct 31 03:32:45.584743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.584710 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.638231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.620140 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e192091-serving-cert\"" Oct 31 03:32:45.668315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.668288 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb"] Oct 31 03:32:45.785050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.784958 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.794513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.794470 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mcb\" (UniqueName: \"kubernetes.io/projected/dd010486-7af5-4b18-969b-7a026a38fc89-kube-api-access-t2mcb\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.812413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.812382 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd010486-7af5-4b18-969b-7a026a38fc89-openshift-service-ca\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.854310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.842651 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-metrics-certs\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.989224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.983775 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd010486-7af5-4b18-969b-7a026a38fc89-openshift-service-ca\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.989224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.984224 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-metrics-certs\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.989224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.984721 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.989224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.985167 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mcb\" (UniqueName: \"kubernetes.io/projected/dd010486-7af5-4b18-969b-7a026a38fc89-kube-api-access-t2mcb\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:45.989224 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:32:45.986124 5520 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-serving-cert: secret "maas-api-e2e-ait-0e192091-serving-cert" not found Oct 31 03:32:45.989224 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:32:45.986220 5520 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls podName:dd010486-7af5-4b18-969b-7a026a38fc89 nodeName:}" failed. No retries permitted until 2026-10-31 03:32:46.486177391 +0000 UTC m=+990.680700494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls") pod "maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" (UID: "dd010486-7af5-4b18-969b-7a026a38fc89") : secret "maas-api-e2e-ait-0e192091-serving-cert" not found Oct 31 03:32:45.989224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.986620 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd010486-7af5-4b18-969b-7a026a38fc89-openshift-service-ca\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:46.004672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:45.998836 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mcb\" (UniqueName: \"kubernetes.io/projected/dd010486-7af5-4b18-969b-7a026a38fc89-kube-api-access-t2mcb\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:46.006274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.006247 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-metrics-certs\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:46.097303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.097141 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d"] Oct 31 03:32:46.113450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.113411 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" podUID="f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" containerName="maas-api" containerID="cri-o://56c042c1bf11d46ccc99560aaf610f986e7a4e32831596a3d5b5ca7a2415157d" gracePeriod=30 Oct 31 03:32:46.335377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.334437 5520 generic.go:356] "Generic (PLEG): container finished" podID="f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" containerID="56c042c1bf11d46ccc99560aaf610f986e7a4e32831596a3d5b5ca7a2415157d" exitCode=0 Oct 31 03:32:46.335377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.334594 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" event={"ID":"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c","Type":"ContainerDied","Data":"56c042c1bf11d46ccc99560aaf610f986e7a4e32831596a3d5b5ca7a2415157d"} Oct 31 03:32:46.518236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.496598 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:46.545790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.545751 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf"] Oct 31 03:32:46.558774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.558729 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" podUID="d683eb31-9988-4255-b83b-de0ef5f032da" containerName="istio-proxy" containerID="cri-o://07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651" gracePeriod=30 Oct 31 03:32:46.612290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.604725 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls\") pod \"maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:46.747227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:46.735968 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:47.326399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.326311 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:47.391281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.391119 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-openshift-service-ca\") pod \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " Oct 31 03:32:47.392650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.392622 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-maas-api-tls\") pod \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " Oct 31 03:32:47.414869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.392252 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-openshift-service-ca" pod "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" (UID: "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:47.414869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.413426 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-kube-api-access-hsb9n\" (UniqueName: \"kubernetes.io/projected/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-kube-api-access-hsb9n\") pod \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " Oct 31 03:32:47.414869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.413632 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-metrics-certs\") pod \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\" (UID: \"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c\") " Oct 31 03:32:47.414869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.414549 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:47.420078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.420028 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" event={"ID":"f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c","Type":"ContainerDied","Data":"ddaf8d102264d3d5d1e639d258538d9087eb7c068fd4f014c3a366bd7af34889"} Oct 31 03:32:47.420625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.420585 5520 scope.go:122] "RemoveContainer" containerID="56c042c1bf11d46ccc99560aaf610f986e7a4e32831596a3d5b5ca7a2415157d" Oct 31 03:32:47.421060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.421043 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" Oct 31 03:32:47.482431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.482390 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-metrics-certs" pod "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" (UID: "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:47.488420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.488360 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-maas-api-tls" pod "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" (UID: "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:47.491029 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.491002 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-kube-api-access-hsb9n" pod "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" (UID: "f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c"). InnerVolumeSpecName "kube-api-access-hsb9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:47.515893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.515698 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:47.515893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.515726 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:47.515893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.515736 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsb9n\" (UniqueName: \"kubernetes.io/projected/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c-kube-api-access-hsb9n\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:47.691878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.691848 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb"] Oct 31 03:32:47.693490 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:32:47.693460 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd010486_7af5_4b18_969b_7a026a38fc89.slice/crio-1f279104397df9a2f8471750c721364ea5a3f372ab378ad8122e4661c58d2502 WatchSource:0}: Error finding container 1f279104397df9a2f8471750c721364ea5a3f372ab378ad8122e4661c58d2502: Status 404 returned error can't find the container with id 1f279104397df9a2f8471750c721364ea5a3f372ab378ad8122e4661c58d2502 Oct 31 03:32:47.744344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.744313 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d"] Oct 31 03:32:47.749605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.749566 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d"] Oct 31 03:32:47.918987 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:47.918952 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" path="/var/lib/kubelet/pods/f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c/volumes" Oct 31 03:32:48.432700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:48.432664 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" event={"ID":"dd010486-7af5-4b18-969b-7a026a38fc89","Type":"ContainerStarted","Data":"3ac5a8c53ff04a38eb388052eb779f445fb9d5335e076e84fcc04b9d038e33e2"} Oct 31 03:32:48.433130 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:48.432721 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" event={"ID":"dd010486-7af5-4b18-969b-7a026a38fc89","Type":"ContainerStarted","Data":"1f279104397df9a2f8471750c721364ea5a3f372ab378ad8122e4661c58d2502"} Oct 31 03:32:48.433724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:48.433701 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:48.454724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:48.454675 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" podStartSLOduration=2.8172602429999998 podStartE2EDuration="3.454657638s" totalImagesPullingTime="637.397395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:48.45312795 +0000 UTC m=+992.647651072" watchObservedRunningTime="2026-10-31 03:32:48.454657638 +0000 UTC m=+992.649180758" Oct 31 03:32:49.771448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:49.771410 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72b46d88-dc8859fc7-2j67d" podUID="f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.87:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:50.140062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.139967 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" podUID="a9ea74c2-7f27-4178-be29-708cb28bb9bd" containerName="payload-processing" containerID="cri-o://dfc809fcb226ac796ca846ce07ed84a0092fc282934973ff4e703bb6a2201d8f" gracePeriod=25 Oct 31 03:32:50.154748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.154714 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" podUID="dce71e50-387a-464a-b741-1a3293043f66" containerName="payload-pre-processing" containerID="cri-o://05ad93d0eacf77a2065fb0845583916accc873b5fee28e6c8aec8e4dfbd1a561" gracePeriod=25 Oct 31 03:32:50.520880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.520604 5520 generic.go:356] "Generic (PLEG): container finished" podID="dce71e50-387a-464a-b741-1a3293043f66" containerID="05ad93d0eacf77a2065fb0845583916accc873b5fee28e6c8aec8e4dfbd1a561" exitCode=0 Oct 31 03:32:50.520880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.520782 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" event={"ID":"dce71e50-387a-464a-b741-1a3293043f66","Type":"ContainerDied","Data":"05ad93d0eacf77a2065fb0845583916accc873b5fee28e6c8aec8e4dfbd1a561"} Oct 31 03:32:50.522942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.522916 5520 generic.go:356] "Generic (PLEG): container finished" podID="a9ea74c2-7f27-4178-be29-708cb28bb9bd" containerID="dfc809fcb226ac796ca846ce07ed84a0092fc282934973ff4e703bb6a2201d8f" exitCode=0 Oct 31 03:32:50.523128 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.523108 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" event={"ID":"a9ea74c2-7f27-4178-be29-708cb28bb9bd","Type":"ContainerDied","Data":"dfc809fcb226ac796ca846ce07ed84a0092fc282934973ff4e703bb6a2201d8f"} Oct 31 03:32:50.565362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.565335 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:50.653455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.653428 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:50.697566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.697535 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce71e50-387a-464a-b741-1a3293043f66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce71e50-387a-464a-b741-1a3293043f66-plugins-config-volume\") pod \"dce71e50-387a-464a-b741-1a3293043f66\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " Oct 31 03:32:50.697743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.697667 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ea74c2-7f27-4178-be29-708cb28bb9bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ea74c2-7f27-4178-be29-708cb28bb9bd-plugins-config-volume\") pod \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " Oct 31 03:32:50.697743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.697716 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce71e50-387a-464a-b741-1a3293043f66-kube-api-access-v6t69\" (UniqueName: \"kubernetes.io/projected/dce71e50-387a-464a-b741-1a3293043f66-kube-api-access-v6t69\") pod \"dce71e50-387a-464a-b741-1a3293043f66\" (UID: \"dce71e50-387a-464a-b741-1a3293043f66\") " Oct 31 03:32:50.697902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.697886 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ea74c2-7f27-4178-be29-708cb28bb9bd-kube-api-access-qr2cp\" (UniqueName: \"kubernetes.io/projected/a9ea74c2-7f27-4178-be29-708cb28bb9bd-kube-api-access-qr2cp\") pod \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\" (UID: \"a9ea74c2-7f27-4178-be29-708cb28bb9bd\") " Oct 31 03:32:50.697963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.697932 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce71e50-387a-464a-b741-1a3293043f66-plugins-config-volume" pod "dce71e50-387a-464a-b741-1a3293043f66" (UID: "dce71e50-387a-464a-b741-1a3293043f66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:50.698450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.698423 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ea74c2-7f27-4178-be29-708cb28bb9bd-plugins-config-volume" pod "a9ea74c2-7f27-4178-be29-708cb28bb9bd" (UID: "a9ea74c2-7f27-4178-be29-708cb28bb9bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:50.698615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.698544 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce71e50-387a-464a-b741-1a3293043f66-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:50.700840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.700795 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce71e50-387a-464a-b741-1a3293043f66-kube-api-access-v6t69" pod "dce71e50-387a-464a-b741-1a3293043f66" (UID: "dce71e50-387a-464a-b741-1a3293043f66"). InnerVolumeSpecName "kube-api-access-v6t69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:50.701484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.701454 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ea74c2-7f27-4178-be29-708cb28bb9bd-kube-api-access-qr2cp" pod "a9ea74c2-7f27-4178-be29-708cb28bb9bd" (UID: "a9ea74c2-7f27-4178-be29-708cb28bb9bd"). InnerVolumeSpecName "kube-api-access-qr2cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:50.800229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.800131 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ea74c2-7f27-4178-be29-708cb28bb9bd-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:50.800229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.800163 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6t69\" (UniqueName: \"kubernetes.io/projected/dce71e50-387a-464a-b741-1a3293043f66-kube-api-access-v6t69\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:50.800229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:50.800173 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr2cp\" (UniqueName: \"kubernetes.io/projected/a9ea74c2-7f27-4178-be29-708cb28bb9bd-kube-api-access-qr2cp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:51.531380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.531353 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" Oct 31 03:32:51.531568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.531397 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6" event={"ID":"dce71e50-387a-464a-b741-1a3293043f66","Type":"ContainerDied","Data":"2cf4d1e132257aec6a710d1d64b8cd61391a891682a45e082ff3c547fe605675"} Oct 31 03:32:51.531568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.531456 5520 scope.go:122] "RemoveContainer" containerID="05ad93d0eacf77a2065fb0845583916accc873b5fee28e6c8aec8e4dfbd1a561" Oct 31 03:32:51.533310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.533034 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" event={"ID":"a9ea74c2-7f27-4178-be29-708cb28bb9bd","Type":"ContainerDied","Data":"33c8da82c9d77a81f20131fc07fc8ef4fbad826285bbe014dd0fc0af94b576a5"} Oct 31 03:32:51.533310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.533054 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n" Oct 31 03:32:51.543026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.543005 5520 scope.go:122] "RemoveContainer" containerID="dfc809fcb226ac796ca846ce07ed84a0092fc282934973ff4e703bb6a2201d8f" Oct 31 03:32:51.559989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.559966 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n"] Oct 31 03:32:51.563881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.563854 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-72b46d88-55ccb577bf-2jj6n"] Oct 31 03:32:51.576073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.576052 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6"] Oct 31 03:32:51.580983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.580957 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-72b46d88-7bf98fc7ff-7ngb6"] Oct 31 03:32:51.920360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.920289 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ea74c2-7f27-4178-be29-708cb28bb9bd" path="/var/lib/kubelet/pods/a9ea74c2-7f27-4178-be29-708cb28bb9bd/volumes" Oct 31 03:32:51.920779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:51.920765 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce71e50-387a-464a-b741-1a3293043f66" path="/var/lib/kubelet/pods/dce71e50-387a-464a-b741-1a3293043f66/volumes" Oct 31 03:32:52.455532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.455503 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:52.542630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.542596 5520 generic.go:356] "Generic (PLEG): container finished" podID="d683eb31-9988-4255-b83b-de0ef5f032da" containerID="07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651" exitCode=0 Oct 31 03:32:52.542842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.542644 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" Oct 31 03:32:52.542842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.542688 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" event={"ID":"d683eb31-9988-4255-b83b-de0ef5f032da","Type":"ContainerDied","Data":"07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651"} Oct 31 03:32:52.542842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.542744 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf" event={"ID":"d683eb31-9988-4255-b83b-de0ef5f032da","Type":"ContainerDied","Data":"9737ae0c583fd4bc3cbf7436375515ec9a7b52b684fe6b2a6c1b347b26cc3263"} Oct 31 03:32:52.542842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.542776 5520 scope.go:122] "RemoveContainer" containerID="07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651" Oct 31 03:32:52.549287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549265 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-data\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549322 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d683eb31-9988-4255-b83b-de0ef5f032da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d683eb31-9988-4255-b83b-de0ef5f032da-istio-podinfo\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549353 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d683eb31-9988-4255-b83b-de0ef5f032da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d683eb31-9988-4255-b83b-de0ef5f032da-istiod-ca-cert\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549428 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-certs\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549473 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-credential-socket\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549521 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-socket\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549545 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-kube-api-access-55qn8\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-kube-api-access-55qn8\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549554 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-data" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:52.549684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549579 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-istio-token\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-istio-token\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549621 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-envoy\") pod \"d683eb31-9988-4255-b83b-de0ef5f032da\" (UID: \"d683eb31-9988-4255-b83b-de0ef5f032da\") " Oct 31 03:32:52.549891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549745 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-certs" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:52.549891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549792 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-credential-socket" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:52.549891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549816 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-socket" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:52.549891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.549854 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d683eb31-9988-4255-b83b-de0ef5f032da-istiod-ca-cert" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:52.550361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.550337 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d683eb31-9988-4255-b83b-de0ef5f032da-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.550451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.550365 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.550451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.550379 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.550451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.550392 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.550451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.550406 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.551934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.551900 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-istio-token" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:52.552258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.552236 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d683eb31-9988-4255-b83b-de0ef5f032da-istio-podinfo" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:32:52.552258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.552242 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-kube-api-access-55qn8" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "kube-api-access-55qn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:52.552384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.552294 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-envoy" pod "d683eb31-9988-4255-b83b-de0ef5f032da" (UID: "d683eb31-9988-4255-b83b-de0ef5f032da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:52.557926 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.554929 5520 scope.go:122] "RemoveContainer" containerID="07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651" Oct 31 03:32:52.557926 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:32:52.555607 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651\": container with ID starting with 07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651 not found: ID does not exist" containerID="07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651" Oct 31 03:32:52.557926 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.555641 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651"} err="failed to get container status \"07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651\": rpc error: code = NotFound desc = could not find container \"07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651\": container with ID starting with 07daf504bd0fecc1e6e2887158722a219f2eb4cfa6a6b669a907ef6529e87651 not found: ID does not exist" Oct 31 03:32:52.651817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.651780 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55qn8\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-kube-api-access-55qn8\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.651817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.651811 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d683eb31-9988-4255-b83b-de0ef5f032da-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.651817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.651820 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d683eb31-9988-4255-b83b-de0ef5f032da-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.652047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.651830 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d683eb31-9988-4255-b83b-de0ef5f032da-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:52.866126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.866093 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf"] Oct 31 03:32:52.872522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:52.872492 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-72b46d88-openshift-default-79bd4b4997-dxmdf"] Oct 31 03:32:53.261021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:53.260992 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:32:53.261549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:53.261069 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:32:53.921463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:53.921421 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d683eb31-9988-4255-b83b-de0ef5f032da" path="/var/lib/kubelet/pods/d683eb31-9988-4255-b83b-de0ef5f032da/volumes" Oct 31 03:32:55.549307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:55.549085 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:32:57.521339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.521301 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5"] Oct 31 03:32:57.522809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.522781 5520 cpu_manager.go:414] "containerMap: removing container" podUID="dce71e50-387a-464a-b741-1a3293043f66" containerName="payload-pre-processing" Oct 31 03:32:57.522971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.522957 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce71e50-387a-464a-b741-1a3293043f66" containerName="payload-pre-processing" Oct 31 03:32:57.523114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523102 5520 cpu_manager.go:414] "containerMap: removing container" podUID="d683eb31-9988-4255-b83b-de0ef5f032da" containerName="istio-proxy" Oct 31 03:32:57.523233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523220 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d683eb31-9988-4255-b83b-de0ef5f032da" containerName="istio-proxy" Oct 31 03:32:57.523358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523348 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" containerName="maas-api" Oct 31 03:32:57.523429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523420 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" containerName="maas-api" Oct 31 03:32:57.523518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523508 5520 cpu_manager.go:414] "containerMap: removing container" podUID="a9ea74c2-7f27-4178-be29-708cb28bb9bd" containerName="payload-processing" Oct 31 03:32:57.523587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523577 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ea74c2-7f27-4178-be29-708cb28bb9bd" containerName="payload-processing" Oct 31 03:32:57.523833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523819 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="d683eb31-9988-4255-b83b-de0ef5f032da" containerName="istio-proxy" Oct 31 03:32:57.523932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.523923 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a8aa76-13c2-42c8-a9ae-d2c474b1d18c" containerName="maas-api" Oct 31 03:32:57.524011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.524002 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ea74c2-7f27-4178-be29-708cb28bb9bd" containerName="payload-processing" Oct 31 03:32:57.524089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.524080 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce71e50-387a-464a-b741-1a3293043f66" containerName="payload-pre-processing" Oct 31 03:32:57.757227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.754449 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5"] Oct 31 03:32:57.757227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.755319 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:57.802941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:57.802853 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50d2d14c-openshift-default-dockercfg-vtnzt\"" Oct 31 03:32:58.018124 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.018090 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.028300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.018489 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.028848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.028818 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.029144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.029122 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.029344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.029323 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.029492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.029467 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.029633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.029616 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.029736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.029719 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhh5\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-kube-api-access-srhh5\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.029914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.029900 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.030581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.027498 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:58.052240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.051296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:58.066315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.060593 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:58.067522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.066583 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:32:58.082979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.079299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:32:58.132611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.132180 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.132955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.132932 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.133936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.133917 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.134648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.134629 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.134936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.134917 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.135506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.135486 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.150229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.136179 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhh5\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-kube-api-access-srhh5\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.152674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.152651 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.153002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.134436 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.153141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.133799 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.153363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.152329 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.162316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.152840 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.162741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.132580 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.162874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.153913 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.163849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.163827 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.190241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.167732 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.244243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.222007 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.274236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.265949 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhh5\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-kube-api-access-srhh5\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.335635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.335542 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:32:58.506772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.506732 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:32:58.781227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.765388 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:32:58.781227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.766480 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:58.781227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.772166 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:58.842537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.842497 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:58.958709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:58.958667 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:32:59.040747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.040651 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.051269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.049542 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:32:59.081287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.055286 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.081287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.055832 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:32:59.093257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.092913 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.342390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.342266 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:32:59.347641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.346825 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:32:59.547520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.532466 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.547520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.533163 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:32:59.574858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.573306 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5"] Oct 31 03:32:59.636033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.635911 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:59.636385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.636349 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:59.636638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.636617 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:32:59.752031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.751939 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.770112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.768907 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.770991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.770966 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:32:59.984915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:32:59.980716 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.024709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.023014 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.034408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.034378 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:33:00.059791 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.058361 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.059791 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.058464 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:33:00.067766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.067723 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.215432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.215403 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:33:00.364276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.363309 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.607239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.581026 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.706898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.706110 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" event={"ID":"25c3c6f6-df1a-4885-96ed-5a840a5f41c6","Type":"ContainerStarted","Data":"8c4a7e96b51370e34d1bcb220d9c66165402fc62d665343cec970e598188ffa9"} Oct 31 03:33:00.707629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.707585 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" event={"ID":"25c3c6f6-df1a-4885-96ed-5a840a5f41c6","Type":"ContainerStarted","Data":"ae5ba28027bf6809f8bf59799a6bf1ee3484c8f235602be11148d5269d204940"} Oct 31 03:33:00.801269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.799568 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.868237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.867612 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podStartSLOduration=3.8675905779999997 podStartE2EDuration="3.867590578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:00.761284002 +0000 UTC m=+1004.955807123" watchObservedRunningTime="2026-10-31 03:33:00.867590578 +0000 UTC m=+1005.062113701" Oct 31 03:33:00.871240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.870415 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:33:00.996081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:00.995518 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:01.076735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.076693 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:01.089577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.089548 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:33:01.108999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.108973 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:33:01.189988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.189920 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:33:01.229119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.229090 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:33:01.336514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.336433 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:01.344666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.344331 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:01.730167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.724988 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:01.732720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:01.731482 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:04.843995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.843963 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj"] Oct 31 03:33:04.848467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.848448 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:04.858893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.858870 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-50d2d14c\"" Oct 31 03:33:04.859967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.859926 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-50d2d14c-dockercfg-qjz6p\"" Oct 31 03:33:04.865168 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.865145 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj"] Oct 31 03:33:04.908461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.908425 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4"] Oct 31 03:33:04.914962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.913913 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:04.925369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.925337 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4"] Oct 31 03:33:05.006287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.998267 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993d5e91-a531-47a9-b3b2-48e66b40f1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:05.006287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.998387 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2b4e6d-7217-43b7-8154-9a417b58a292-plugins-config-volume\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:05.006287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.998440 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2czm\" (UniqueName: \"kubernetes.io/projected/da2b4e6d-7217-43b7-8154-9a417b58a292-kube-api-access-b2czm\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:05.006287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:04.998588 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf44t\" (UniqueName: \"kubernetes.io/projected/993d5e91-a531-47a9-b3b2-48e66b40f1ac-kube-api-access-xf44t\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:05.125279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.120629 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2b4e6d-7217-43b7-8154-9a417b58a292-plugins-config-volume\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:05.125279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.120710 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2czm\" (UniqueName: \"kubernetes.io/projected/da2b4e6d-7217-43b7-8154-9a417b58a292-kube-api-access-b2czm\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:05.135232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.132617 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf44t\" (UniqueName: \"kubernetes.io/projected/993d5e91-a531-47a9-b3b2-48e66b40f1ac-kube-api-access-xf44t\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:05.135232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.132913 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993d5e91-a531-47a9-b3b2-48e66b40f1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:05.135232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.133958 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993d5e91-a531-47a9-b3b2-48e66b40f1ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:05.135232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.134787 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2b4e6d-7217-43b7-8154-9a417b58a292-plugins-config-volume\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:05.370227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.363438 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf44t\" (UniqueName: \"kubernetes.io/projected/993d5e91-a531-47a9-b3b2-48e66b40f1ac-kube-api-access-xf44t\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:05.462517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.462484 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:05.501191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.501162 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2czm\" (UniqueName: \"kubernetes.io/projected/da2b4e6d-7217-43b7-8154-9a417b58a292-kube-api-access-b2czm\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:05.699637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.699603 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:05.805682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.805648 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj"] Oct 31 03:33:05.806776 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:05.806245 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993d5e91_a531_47a9_b3b2_48e66b40f1ac.slice/crio-738012399e9fd881b9d8fa56511974d118507b953ac630fb63ae6f76228aaae0 WatchSource:0}: Error finding container 738012399e9fd881b9d8fa56511974d118507b953ac630fb63ae6f76228aaae0: Status 404 returned error can't find the container with id 738012399e9fd881b9d8fa56511974d118507b953ac630fb63ae6f76228aaae0 Oct 31 03:33:05.985333 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:05.985297 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2b4e6d_7217_43b7_8154_9a417b58a292.slice/crio-8faff16e7b97661f92141ca39bda89b325e34e85e30ae7a96d4514a9094a6107 WatchSource:0}: Error finding container 8faff16e7b97661f92141ca39bda89b325e34e85e30ae7a96d4514a9094a6107: Status 404 returned error can't find the container with id 8faff16e7b97661f92141ca39bda89b325e34e85e30ae7a96d4514a9094a6107 Oct 31 03:33:05.985705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:05.985678 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4"] Oct 31 03:33:06.766577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.766538 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" event={"ID":"da2b4e6d-7217-43b7-8154-9a417b58a292","Type":"ContainerStarted","Data":"5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c"} Oct 31 03:33:06.766757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.766598 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" event={"ID":"da2b4e6d-7217-43b7-8154-9a417b58a292","Type":"ContainerStarted","Data":"8faff16e7b97661f92141ca39bda89b325e34e85e30ae7a96d4514a9094a6107"} Oct 31 03:33:06.766757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.766734 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:06.768255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.768230 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" event={"ID":"993d5e91-a531-47a9-b3b2-48e66b40f1ac","Type":"ContainerStarted","Data":"9af914b16a13e41f65d4cc70119a3e41e81cae92ad7b65faee0cb5baa0ee2d3e"} Oct 31 03:33:06.768371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.768266 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" event={"ID":"993d5e91-a531-47a9-b3b2-48e66b40f1ac","Type":"ContainerStarted","Data":"738012399e9fd881b9d8fa56511974d118507b953ac630fb63ae6f76228aaae0"} Oct 31 03:33:06.768445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.768371 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:06.783531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.783486 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" podStartSLOduration=2.7834733419999997 podStartE2EDuration="2.783473342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:06.782102633 +0000 UTC m=+1010.976625776" watchObservedRunningTime="2026-10-31 03:33:06.783473342 +0000 UTC m=+1010.977996463" Oct 31 03:33:06.801794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:06.801723 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" podStartSLOduration=2.801706319 podStartE2EDuration="2.801706319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:06.796299696 +0000 UTC m=+1010.990822829" watchObservedRunningTime="2026-10-31 03:33:06.801706319 +0000 UTC m=+1010.996229441" Oct 31 03:33:09.359632 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:33:09.359591 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:33:10.088945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.088913 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd"] Oct 31 03:33:10.098323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.097250 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.114537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.114507 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-50d2d14c-serving-cert\"" Oct 31 03:33:10.115978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.115948 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd"] Oct 31 03:33:10.121630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.121330 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzts\" (UniqueName: \"kubernetes.io/projected/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-kube-api-access-zvzts\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.121630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.121480 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.121630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.121581 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-metrics-certs\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.121819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.121708 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-openshift-service-ca\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.240066 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.229495 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.240066 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.229615 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-metrics-certs\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.240066 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.229740 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-openshift-service-ca\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.240066 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.229873 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzts\" (UniqueName: \"kubernetes.io/projected/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-kube-api-access-zvzts\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.263228 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:33:10.248819 5520 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-serving-cert: secret "maas-api-e2e-mt-50d2d14c-serving-cert" not found Oct 31 03:33:10.285225 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:33:10.281313 5520 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls podName:97bd4c3a-aaf9-4b7f-ab96-7017555a32e6 nodeName:}" failed. No retries permitted until 2026-10-31 03:33:10.781283275 +0000 UTC m=+1014.975806385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls") pod "maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" (UID: "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6") : secret "maas-api-e2e-mt-50d2d14c-serving-cert" not found Oct 31 03:33:10.285225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.251765 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-openshift-service-ca\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.296812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.296779 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzts\" (UniqueName: \"kubernetes.io/projected/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-kube-api-access-zvzts\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.298774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.298631 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-metrics-certs\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.849964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.849927 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:10.882231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:10.873617 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:11.032894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:11.030950 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:11.353725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:11.353692 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd"] Oct 31 03:33:11.354185 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:11.354160 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bd4c3a_aaf9_4b7f_ab96_7017555a32e6.slice/crio-929da85ee08c00c13029b7a7573eedc21715eee8cb0d3a56c9625a8df11eaab5 WatchSource:0}: Error finding container 929da85ee08c00c13029b7a7573eedc21715eee8cb0d3a56c9625a8df11eaab5: Status 404 returned error can't find the container with id 929da85ee08c00c13029b7a7573eedc21715eee8cb0d3a56c9625a8df11eaab5 Oct 31 03:33:11.846890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:11.846860 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" event={"ID":"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6","Type":"ContainerStarted","Data":"929da85ee08c00c13029b7a7573eedc21715eee8cb0d3a56c9625a8df11eaab5"} Oct 31 03:33:12.872333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:12.872297 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" event={"ID":"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6","Type":"ContainerStarted","Data":"594465c4821bd6fc4f740189825d8d14813f1f1430ba4cc4dec6960b59397c13"} Oct 31 03:33:12.874377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:12.874351 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:12.897995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:12.897942 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" podStartSLOduration=2.414868895 podStartE2EDuration="2.897927987s" totalImagesPullingTime="483.059092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:12.896638704 +0000 UTC m=+1017.091161825" watchObservedRunningTime="2026-10-31 03:33:12.897927987 +0000 UTC m=+1017.092451102" Oct 31 03:33:15.398076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:15.398041 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv"] Oct 31 03:33:15.411578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:15.411549 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c"] Oct 31 03:33:17.800543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:17.800513 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:33:17.828231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:17.827361 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:33:19.898084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:19.898050 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:33:20.400327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.400272 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" podUID="ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" containerName="payload-processing" containerID="cri-o://de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791" gracePeriod=25 Oct 31 03:33:20.412381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.412351 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" podUID="054e4608-7e1c-431f-a929-1c99c2d13456" containerName="payload-pre-processing" containerID="cri-o://3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848" gracePeriod=25 Oct 31 03:33:20.756328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.756296 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:33:20.840023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.838550 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-plugins-config-volume\") pod \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " Oct 31 03:33:20.864324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.862446 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-plugins-config-volume" pod "ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" (UID: "ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:20.942713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.940568 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-kube-api-access-kpsdl\" (UniqueName: \"kubernetes.io/projected/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-kube-api-access-kpsdl\") pod \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\" (UID: \"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a\") " Oct 31 03:33:20.942713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.941277 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:20.946045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.945970 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-kube-api-access-kpsdl" pod "ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" (UID: "ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a"). InnerVolumeSpecName "kube-api-access-kpsdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:20.979165 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:20.979135 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:33:21.042019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.041989 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054e4608-7e1c-431f-a929-1c99c2d13456-kube-api-access-j926r\" (UniqueName: \"kubernetes.io/projected/054e4608-7e1c-431f-a929-1c99c2d13456-kube-api-access-j926r\") pod \"054e4608-7e1c-431f-a929-1c99c2d13456\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " Oct 31 03:33:21.042188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.042062 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054e4608-7e1c-431f-a929-1c99c2d13456-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054e4608-7e1c-431f-a929-1c99c2d13456-plugins-config-volume\") pod \"054e4608-7e1c-431f-a929-1c99c2d13456\" (UID: \"054e4608-7e1c-431f-a929-1c99c2d13456\") " Oct 31 03:33:21.042541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.042520 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpsdl\" (UniqueName: \"kubernetes.io/projected/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a-kube-api-access-kpsdl\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:21.042844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.042822 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054e4608-7e1c-431f-a929-1c99c2d13456-plugins-config-volume" pod "054e4608-7e1c-431f-a929-1c99c2d13456" (UID: "054e4608-7e1c-431f-a929-1c99c2d13456"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:21.044535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.044513 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054e4608-7e1c-431f-a929-1c99c2d13456-kube-api-access-j926r" pod "054e4608-7e1c-431f-a929-1c99c2d13456" (UID: "054e4608-7e1c-431f-a929-1c99c2d13456"). InnerVolumeSpecName "kube-api-access-j926r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:21.114607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.114577 5520 generic.go:356] "Generic (PLEG): container finished" podID="054e4608-7e1c-431f-a929-1c99c2d13456" containerID="3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848" exitCode=0 Oct 31 03:33:21.114890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.114626 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" Oct 31 03:33:21.114974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.114649 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" event={"ID":"054e4608-7e1c-431f-a929-1c99c2d13456","Type":"ContainerDied","Data":"3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848"} Oct 31 03:33:21.115034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.115011 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c" event={"ID":"054e4608-7e1c-431f-a929-1c99c2d13456","Type":"ContainerDied","Data":"240e19c49c7306e6b37f49bffaac91d3958d2601d1804c3634c4c7a59951b8e7"} Oct 31 03:33:21.115082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.115046 5520 scope.go:122] "RemoveContainer" containerID="3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848" Oct 31 03:33:21.117774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.117554 5520 generic.go:356] "Generic (PLEG): container finished" podID="ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" containerID="de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791" exitCode=0 Oct 31 03:33:21.117774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.117603 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" Oct 31 03:33:21.117774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.117695 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" event={"ID":"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a","Type":"ContainerDied","Data":"de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791"} Oct 31 03:33:21.117774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.117738 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv" event={"ID":"ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a","Type":"ContainerDied","Data":"918862be37851c3ba9cb6b9bfea8f184a0ab3588c7c8cc749a7e948a74a924d9"} Oct 31 03:33:21.128736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.128711 5520 scope.go:122] "RemoveContainer" containerID="3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848" Oct 31 03:33:21.129025 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:33:21.129004 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848\": container with ID starting with 3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848 not found: ID does not exist" containerID="3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848" Oct 31 03:33:21.129095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.129038 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848"} err="failed to get container status \"3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848\": rpc error: code = NotFound desc = could not find container \"3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848\": container with ID starting with 3ffd3d2021884b4b88a3e6a0cb4419517b4ff077fa2b49270df7311627158848 not found: ID does not exist" Oct 31 03:33:21.129095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.129055 5520 scope.go:122] "RemoveContainer" containerID="de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791" Oct 31 03:33:21.141803 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.141777 5520 scope.go:122] "RemoveContainer" containerID="de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791" Oct 31 03:33:21.142170 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:33:21.142148 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791\": container with ID starting with de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791 not found: ID does not exist" containerID="de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791" Oct 31 03:33:21.142368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.142177 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791"} err="failed to get container status \"de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791\": rpc error: code = NotFound desc = could not find container \"de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791\": container with ID starting with de01bbf080c0a7b694b11c26b7cec7d9bf068d2f289ae9b8f30ccb5e74124791 not found: ID does not exist" Oct 31 03:33:21.144042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.144023 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j926r\" (UniqueName: \"kubernetes.io/projected/054e4608-7e1c-431f-a929-1c99c2d13456-kube-api-access-j926r\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:21.144102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.144046 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/054e4608-7e1c-431f-a929-1c99c2d13456-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:21.145951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.145928 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c"] Oct 31 03:33:21.150113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.150092 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e192091-5c4bcf4967-bdn7c"] Oct 31 03:33:21.160264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.160243 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv"] Oct 31 03:33:21.164780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.164761 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e192091-5946b58f57-57kcv"] Oct 31 03:33:21.250647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.250548 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb"] Oct 31 03:33:21.251099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.251068 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" podUID="dd010486-7af5-4b18-969b-7a026a38fc89" containerName="maas-api" containerID="cri-o://3ac5a8c53ff04a38eb388052eb779f445fb9d5335e076e84fcc04b9d038e33e2" gracePeriod=30 Oct 31 03:33:21.990279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.973450 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054e4608-7e1c-431f-a929-1c99c2d13456" path="/var/lib/kubelet/pods/054e4608-7e1c-431f-a929-1c99c2d13456/volumes" Oct 31 03:33:21.990279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:21.974168 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" path="/var/lib/kubelet/pods/ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a/volumes" Oct 31 03:33:22.130226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.130023 5520 generic.go:356] "Generic (PLEG): container finished" podID="dd010486-7af5-4b18-969b-7a026a38fc89" containerID="3ac5a8c53ff04a38eb388052eb779f445fb9d5335e076e84fcc04b9d038e33e2" exitCode=0 Oct 31 03:33:22.130226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.130094 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" event={"ID":"dd010486-7af5-4b18-969b-7a026a38fc89","Type":"ContainerDied","Data":"3ac5a8c53ff04a38eb388052eb779f445fb9d5335e076e84fcc04b9d038e33e2"} Oct 31 03:33:22.201630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.201597 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:33:22.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.253364 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-metrics-certs\") pod \"dd010486-7af5-4b18-969b-7a026a38fc89\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " Oct 31 03:33:22.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.253425 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd010486-7af5-4b18-969b-7a026a38fc89-kube-api-access-t2mcb\" (UniqueName: \"kubernetes.io/projected/dd010486-7af5-4b18-969b-7a026a38fc89-kube-api-access-t2mcb\") pod \"dd010486-7af5-4b18-969b-7a026a38fc89\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " Oct 31 03:33:22.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.253478 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls\") pod \"dd010486-7af5-4b18-969b-7a026a38fc89\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " Oct 31 03:33:22.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.253604 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd010486-7af5-4b18-969b-7a026a38fc89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd010486-7af5-4b18-969b-7a026a38fc89-openshift-service-ca\") pod \"dd010486-7af5-4b18-969b-7a026a38fc89\" (UID: \"dd010486-7af5-4b18-969b-7a026a38fc89\") " Oct 31 03:33:22.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.254473 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd010486-7af5-4b18-969b-7a026a38fc89-openshift-service-ca" pod "dd010486-7af5-4b18-969b-7a026a38fc89" (UID: "dd010486-7af5-4b18-969b-7a026a38fc89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:22.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.256614 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-metrics-certs" pod "dd010486-7af5-4b18-969b-7a026a38fc89" (UID: "dd010486-7af5-4b18-969b-7a026a38fc89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:33:22.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.257549 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls" pod "dd010486-7af5-4b18-969b-7a026a38fc89" (UID: "dd010486-7af5-4b18-969b-7a026a38fc89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:33:22.263776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.263745 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd010486-7af5-4b18-969b-7a026a38fc89-kube-api-access-t2mcb" pod "dd010486-7af5-4b18-969b-7a026a38fc89" (UID: "dd010486-7af5-4b18-969b-7a026a38fc89"). InnerVolumeSpecName "kube-api-access-t2mcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:22.356176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.356134 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:22.356176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.356178 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2mcb\" (UniqueName: \"kubernetes.io/projected/dd010486-7af5-4b18-969b-7a026a38fc89-kube-api-access-t2mcb\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:22.356449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.356194 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd010486-7af5-4b18-969b-7a026a38fc89-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:22.356449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:22.356234 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd010486-7af5-4b18-969b-7a026a38fc89-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:23.125933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.125889 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl"] Oct 31 03:33:23.126506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.126154 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" podUID="5e25310c-5979-447d-9dcc-678bef6dbfae" containerName="istio-proxy" containerID="cri-o://8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238" gracePeriod=30 Oct 31 03:33:23.146887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.146846 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" event={"ID":"dd010486-7af5-4b18-969b-7a026a38fc89","Type":"ContainerDied","Data":"1f279104397df9a2f8471750c721364ea5a3f372ab378ad8122e4661c58d2502"} Oct 31 03:33:23.147095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.146930 5520 scope.go:122] "RemoveContainer" containerID="3ac5a8c53ff04a38eb388052eb779f445fb9d5335e076e84fcc04b9d038e33e2" Oct 31 03:33:23.147169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.147099 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb" Oct 31 03:33:23.181959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.181919 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb"] Oct 31 03:33:23.194365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.194335 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e192091-7c4d67cb9b-l6jvb"] Oct 31 03:33:23.974512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:23.974476 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd010486-7af5-4b18-969b-7a026a38fc89" path="/var/lib/kubelet/pods/dd010486-7af5-4b18-969b-7a026a38fc89/volumes" Oct 31 03:33:24.253303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.245323 5520 patch_prober.go:29] interesting pod/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:33:24.253303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.245395 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" podUID="5e25310c-5979-447d-9dcc-678bef6dbfae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:24.354752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.354711 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs"] Oct 31 03:33:24.355807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.355779 5520 cpu_manager.go:414] "containerMap: removing container" podUID="054e4608-7e1c-431f-a929-1c99c2d13456" containerName="payload-pre-processing" Oct 31 03:33:24.355807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.355807 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054e4608-7e1c-431f-a929-1c99c2d13456" containerName="payload-pre-processing" Oct 31 03:33:24.356015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.355824 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" containerName="payload-processing" Oct 31 03:33:24.356015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.355833 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" containerName="payload-processing" Oct 31 03:33:24.356015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.355872 5520 cpu_manager.go:414] "containerMap: removing container" podUID="dd010486-7af5-4b18-969b-7a026a38fc89" containerName="maas-api" Oct 31 03:33:24.356015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.355880 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd010486-7af5-4b18-969b-7a026a38fc89" containerName="maas-api" Oct 31 03:33:24.356237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.356038 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="054e4608-7e1c-431f-a929-1c99c2d13456" containerName="payload-pre-processing" Oct 31 03:33:24.356237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.356053 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8e94bf-bb8e-43d0-b3e3-6f00dda2316a" containerName="payload-processing" Oct 31 03:33:24.356237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.356068 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd010486-7af5-4b18-969b-7a026a38fc89" containerName="maas-api" Oct 31 03:33:24.361662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.361640 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.371769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.371734 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs"] Oct 31 03:33:24.412603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.412555 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fc5ece-openshift-default-dockercfg-n4n4f\"" Oct 31 03:33:24.494228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.454326 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnn2\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-kube-api-access-tqnn2\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.494228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.454387 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-envoy\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.494228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.454434 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-data\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.519072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.518313 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istiod-ca-cert\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.519072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.518493 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-socket\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.519072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.518552 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-certs\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.519072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.518628 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-token\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.519072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.518703 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-credential-socket\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.519072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.518746 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-podinfo\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624254 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-socket\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624326 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-certs\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624397 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-token\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624458 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-credential-socket\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624499 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-podinfo\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624600 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnn2\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-kube-api-access-tqnn2\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624656 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-envoy\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624703 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-data\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.624794 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istiod-ca-cert\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.625658 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istiod-ca-cert\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.625922 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-socket\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.626328 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-certs\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.626854 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-credential-socket\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.628445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.628383 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-data\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.632516 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.632469 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-podinfo\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.642228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.638192 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-token\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.645126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.645073 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-envoy\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.656932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.656873 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnn2\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-kube-api-access-tqnn2\") pod \"e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:24.697816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:24.697491 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:25.311048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:25.311007 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs"] Oct 31 03:33:25.320278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:25.320170 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:33:25.320808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:25.320784 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:33:25.320981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:25.320965 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:33:26.225422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:26.225377 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" event={"ID":"3a1d47d5-5821-40d5-8a86-ef63a1148ae6","Type":"ContainerStarted","Data":"4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1"} Oct 31 03:33:26.225630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:26.225487 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" event={"ID":"3a1d47d5-5821-40d5-8a86-ef63a1148ae6","Type":"ContainerStarted","Data":"64c8f88005b70b3003dbbc1565cab348e600c47d290d9a060f367a90983f80c0"} Oct 31 03:33:26.249275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:26.249218 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podStartSLOduration=2.249190187 podStartE2EDuration="2.249190187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:26.246690247 +0000 UTC m=+1030.441213372" watchObservedRunningTime="2026-10-31 03:33:26.249190187 +0000 UTC m=+1030.443713308" Oct 31 03:33:26.698282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:26.698250 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:26.704859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:26.704833 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:27.233636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:27.233605 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:27.234594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:27.234572 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:33:28.429236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.429218 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:33:28.452544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452071 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-envoy\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.452544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452165 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-token\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-token\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.452544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452355 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-data\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.452544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452396 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-credential-socket\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.452544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452523 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-podinfo\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.453008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452634 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-socket\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.453008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452672 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e25310c-5979-447d-9dcc-678bef6dbfae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25310c-5979-447d-9dcc-678bef6dbfae-istiod-ca-cert\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.453008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452698 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-kube-api-access-8hgdt\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-kube-api-access-8hgdt\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.453008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.452764 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-certs\") pod \"5e25310c-5979-447d-9dcc-678bef6dbfae\" (UID: \"5e25310c-5979-447d-9dcc-678bef6dbfae\") " Oct 31 03:33:28.454378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.454332 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-socket" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:28.454517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.454390 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e25310c-5979-447d-9dcc-678bef6dbfae-istiod-ca-cert" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:33:28.454517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.454403 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-certs" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:28.454676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.454659 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-credential-socket" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:28.454733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.454676 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-data" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:28.455326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.455299 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-envoy" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:33:28.455786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.455759 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-podinfo" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:33:28.457090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.457062 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-kube-api-access-8hgdt" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "kube-api-access-8hgdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:28.459488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.459463 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-token" pod "5e25310c-5979-447d-9dcc-678bef6dbfae" (UID: "5e25310c-5979-447d-9dcc-678bef6dbfae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:33:28.555769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555734 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.555769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555764 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.555769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555774 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.556010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555783 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.556010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555791 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e25310c-5979-447d-9dcc-678bef6dbfae-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.556010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555800 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgdt\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-kube-api-access-8hgdt\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.556010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555809 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.556010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555817 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:28.556010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:28.555825 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e25310c-5979-447d-9dcc-678bef6dbfae-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:33:29.251166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.251127 5520 generic.go:356] "Generic (PLEG): container finished" podID="5e25310c-5979-447d-9dcc-678bef6dbfae" containerID="8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238" exitCode=0 Oct 31 03:33:29.251364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.251195 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" Oct 31 03:33:29.251364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.251234 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" event={"ID":"5e25310c-5979-447d-9dcc-678bef6dbfae","Type":"ContainerDied","Data":"8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238"} Oct 31 03:33:29.251364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.251318 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl" event={"ID":"5e25310c-5979-447d-9dcc-678bef6dbfae","Type":"ContainerDied","Data":"80165d1cd40965d7c5a303cda99666221236999d5737c403dc0adbb947b46f2e"} Oct 31 03:33:29.251364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.251352 5520 scope.go:122] "RemoveContainer" containerID="8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238" Oct 31 03:33:29.262810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.262792 5520 scope.go:122] "RemoveContainer" containerID="8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238" Oct 31 03:33:29.263112 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:33:29.263092 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238\": container with ID starting with 8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238 not found: ID does not exist" containerID="8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238" Oct 31 03:33:29.263174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.263120 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238"} err="failed to get container status \"8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238\": rpc error: code = NotFound desc = could not find container \"8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238\": container with ID starting with 8bd89f3aaa265499b0903424e0330b83e8af2558c8bfa1341159b84a0c301238 not found: ID does not exist" Oct 31 03:33:29.279685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.279661 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl"] Oct 31 03:33:29.285073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.285046 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e192091-openshift-default-67dfdcdf9-v9lnl"] Oct 31 03:33:29.925617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.925579 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e25310c-5979-447d-9dcc-678bef6dbfae" path="/var/lib/kubelet/pods/5e25310c-5979-447d-9dcc-678bef6dbfae/volumes" Oct 31 03:33:29.926097 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.926011 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z"] Oct 31 03:33:29.927014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.926994 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5e25310c-5979-447d-9dcc-678bef6dbfae" containerName="istio-proxy" Oct 31 03:33:29.927084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.927018 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e25310c-5979-447d-9dcc-678bef6dbfae" containerName="istio-proxy" Oct 31 03:33:29.927178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.927165 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e25310c-5979-447d-9dcc-678bef6dbfae" containerName="istio-proxy" Oct 31 03:33:29.932429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.932408 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv"] Oct 31 03:33:29.932560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.932531 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:29.936488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.936468 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:29.941439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.941380 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fc5ece\"" Oct 31 03:33:29.941582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.941541 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-fc5ece-dockercfg-ksrpb\"" Oct 31 03:33:29.947354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.947330 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv"] Oct 31 03:33:29.952413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:29.952389 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z"] Oct 31 03:33:30.039235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.033270 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:30.039235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.033602 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:30.039235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.033727 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhxd\" (UniqueName: \"kubernetes.io/projected/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-kube-api-access-kbhxd\") pod \"payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:30.039235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.033846 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdh9\" (UniqueName: \"kubernetes.io/projected/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-kube-api-access-vvdh9\") pod \"payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:30.182283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.179539 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:30.182283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.179650 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhxd\" (UniqueName: \"kubernetes.io/projected/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-kube-api-access-kbhxd\") pod \"payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:30.182283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.179754 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdh9\" (UniqueName: \"kubernetes.io/projected/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-kube-api-access-vvdh9\") pod \"payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:30.182283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.179813 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:30.182283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.180697 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:30.182283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.181393 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:30.248273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.233153 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhxd\" (UniqueName: \"kubernetes.io/projected/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-kube-api-access-kbhxd\") pod \"payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:30.248273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.234256 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdh9\" (UniqueName: \"kubernetes.io/projected/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-kube-api-access-vvdh9\") pod \"payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:30.270856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.270827 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:30.271502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.271481 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:30.776836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.776788 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z"] Oct 31 03:33:30.776992 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:30.776973 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25e37b8_2e60_4c2a_bc75_5a9e133a8269.slice/crio-7cd29e320df8e29aa33f99b2ef3e7a48a6021a9aedbb0a36a1ef6a1b96ddfb8a WatchSource:0}: Error finding container 7cd29e320df8e29aa33f99b2ef3e7a48a6021a9aedbb0a36a1ef6a1b96ddfb8a: Status 404 returned error can't find the container with id 7cd29e320df8e29aa33f99b2ef3e7a48a6021a9aedbb0a36a1ef6a1b96ddfb8a Oct 31 03:33:30.909730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:30.909699 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv"] Oct 31 03:33:30.910468 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:30.910444 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1381bbc_d77c_42c2_8ab7_d536c4251d1c.slice/crio-a3a6a318227d78d81465fff65d3a2f41959ad0aa4455c44d995a92415133d80e WatchSource:0}: Error finding container a3a6a318227d78d81465fff65d3a2f41959ad0aa4455c44d995a92415133d80e: Status 404 returned error can't find the container with id a3a6a318227d78d81465fff65d3a2f41959ad0aa4455c44d995a92415133d80e Oct 31 03:33:31.384546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.384390 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" event={"ID":"f25e37b8-2e60-4c2a-bc75-5a9e133a8269","Type":"ContainerStarted","Data":"158a89e5807deddf646398c335a860717564b472ace0481843e121f835586b9b"} Oct 31 03:33:31.384546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.384459 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" event={"ID":"f25e37b8-2e60-4c2a-bc75-5a9e133a8269","Type":"ContainerStarted","Data":"7cd29e320df8e29aa33f99b2ef3e7a48a6021a9aedbb0a36a1ef6a1b96ddfb8a"} Oct 31 03:33:31.385123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.384718 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:31.386894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.386862 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" event={"ID":"c1381bbc-d77c-42c2-8ab7-d536c4251d1c","Type":"ContainerStarted","Data":"e738fa93fa58df2ace99e80db6173b201b0d4f4a343fe3561f9b2fe2347292cd"} Oct 31 03:33:31.387111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.386915 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" event={"ID":"c1381bbc-d77c-42c2-8ab7-d536c4251d1c","Type":"ContainerStarted","Data":"a3a6a318227d78d81465fff65d3a2f41959ad0aa4455c44d995a92415133d80e"} Oct 31 03:33:31.387111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.386954 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:31.405065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.405006 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" podStartSLOduration=2.4049905799999998 podStartE2EDuration="2.40499058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:31.398892654 +0000 UTC m=+1035.593415777" watchObservedRunningTime="2026-10-31 03:33:31.40499058 +0000 UTC m=+1035.599513702" Oct 31 03:33:31.416878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:31.416830 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" podStartSLOduration=2.416815624 podStartE2EDuration="2.416815624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:31.4136613 +0000 UTC m=+1035.608184439" watchObservedRunningTime="2026-10-31 03:33:31.416815624 +0000 UTC m=+1035.611338745" Oct 31 03:33:32.427837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.427808 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8"] Oct 31 03:33:32.434177 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.434151 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8"] Oct 31 03:33:32.434360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.434290 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.438656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.438573 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-50d2d14c\"/\"kube-root-ca.crt\"" Oct 31 03:33:32.438656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.438575 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-50d2d14c\"/\"openshift-service-ca.crt\"" Oct 31 03:33:32.439818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.439694 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-50d2d14c\"/\"default-dockercfg-fr5st\"" Oct 31 03:33:32.439818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.439755 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-50d2d14c\"/\"e2e-lifecycle-model-50d2d14c-kserve-self-signed-certs\"" Oct 31 03:33:32.467964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.467931 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tls-certs\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.468155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.468081 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-dshm\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.468360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.468341 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tmp-dir\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.468536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.468504 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-home\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.468714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.468696 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54gs\" (UniqueName: \"kubernetes.io/projected/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-kube-api-access-w54gs\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.469008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.468976 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-model-cache\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.577268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.571455 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-dshm\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.577268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.571782 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tmp-dir\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.577268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.571828 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-home\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.577268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.571881 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54gs\" (UniqueName: \"kubernetes.io/projected/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-kube-api-access-w54gs\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.577268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.571974 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-model-cache\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.577268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.572041 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tls-certs\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.577268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.574601 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-home\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.579021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.578581 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tmp-dir\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.579021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.578940 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-model-cache\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.597631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.596345 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-dshm\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.628990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.616175 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tls-certs\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.648686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.648620 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54gs\" (UniqueName: \"kubernetes.io/projected/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-kube-api-access-w54gs\") pod \"e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:32.757228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:32.755397 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:33.203576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:33.203526 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8"] Oct 31 03:33:33.209574 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:33.209542 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a50bdd8_10c1_47d2_b43f_91274a06c4f9.slice/crio-00cc76f39cf844d72cb13fef3fc273c5eb486c7e72330ddc81f3729c49afd133 WatchSource:0}: Error finding container 00cc76f39cf844d72cb13fef3fc273c5eb486c7e72330ddc81f3729c49afd133: Status 404 returned error can't find the container with id 00cc76f39cf844d72cb13fef3fc273c5eb486c7e72330ddc81f3729c49afd133 Oct 31 03:33:33.407976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:33.407931 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" event={"ID":"8a50bdd8-10c1-47d2-b43f-91274a06c4f9","Type":"ContainerStarted","Data":"c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8"} Oct 31 03:33:33.408166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:33.407999 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" event={"ID":"8a50bdd8-10c1-47d2-b43f-91274a06c4f9","Type":"ContainerStarted","Data":"00cc76f39cf844d72cb13fef3fc273c5eb486c7e72330ddc81f3729c49afd133"} Oct 31 03:33:33.426543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:33.426491 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" podStartSLOduration=1.426475104 podStartE2EDuration="1.426475104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:33.423931569 +0000 UTC m=+1037.618454703" watchObservedRunningTime="2026-10-31 03:33:33.426475104 +0000 UTC m=+1037.620998224" Oct 31 03:33:33.756280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:33.756247 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:34.422460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.422430 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc"] Oct 31 03:33:34.446493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.446456 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.450553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.450526 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-fc5ece-serving-cert\"" Oct 31 03:33:34.452460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.452437 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc"] Oct 31 03:33:34.584222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.579906 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklcv\" (UniqueName: \"kubernetes.io/projected/82b5e11c-6318-46e4-aa59-fb0d7b972691-kube-api-access-qklcv\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.584222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.580017 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.584222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.580194 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b5e11c-6318-46e4-aa59-fb0d7b972691-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.584222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.580430 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.694452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.694418 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklcv\" (UniqueName: \"kubernetes.io/projected/82b5e11c-6318-46e4-aa59-fb0d7b972691-kube-api-access-qklcv\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.694637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.694483 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.694637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.694590 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b5e11c-6318-46e4-aa59-fb0d7b972691-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.694774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.694702 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.695823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.695794 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b5e11c-6318-46e4-aa59-fb0d7b972691-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.697336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.697316 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.697488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.697467 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.702752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.702723 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklcv\" (UniqueName: \"kubernetes.io/projected/82b5e11c-6318-46e4-aa59-fb0d7b972691-kube-api-access-qklcv\") pod \"maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:34.772051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:34.772012 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:35.191280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:35.189116 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc"] Oct 31 03:33:35.277822 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:35.277781 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b5e11c_6318_46e4_aa59_fb0d7b972691.slice/crio-65b84e1b277943cc695a7ee25f01abd98e5f865a0cf8bb9af45813eb60233e97 WatchSource:0}: Error finding container 65b84e1b277943cc695a7ee25f01abd98e5f865a0cf8bb9af45813eb60233e97: Status 404 returned error can't find the container with id 65b84e1b277943cc695a7ee25f01abd98e5f865a0cf8bb9af45813eb60233e97 Oct 31 03:33:35.485483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:35.485400 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" event={"ID":"82b5e11c-6318-46e4-aa59-fb0d7b972691","Type":"ContainerStarted","Data":"65b84e1b277943cc695a7ee25f01abd98e5f865a0cf8bb9af45813eb60233e97"} Oct 31 03:33:36.494889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:36.494847 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" event={"ID":"82b5e11c-6318-46e4-aa59-fb0d7b972691","Type":"ContainerStarted","Data":"c687756912fdbb040f60813c0c15fe85b64745df2ae2e8ac5b09d65a78dbffa0"} Oct 31 03:33:36.495487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:36.494993 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:36.515037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:36.514985 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" podStartSLOduration=2.049288903 podStartE2EDuration="2.514969472s" totalImagesPullingTime="465.680569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:36.513291523 +0000 UTC m=+1040.707814662" watchObservedRunningTime="2026-10-31 03:33:36.514969472 +0000 UTC m=+1040.709492647" Oct 31 03:33:39.360870 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:33:39.360830 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:33:42.403366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:42.403332 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:33:42.404487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:42.404462 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:33:42.531143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:42.531115 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:33:42.756335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:42.756305 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:42.849228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:42.842640 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:43.766156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:43.766121 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:33:47.766389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:47.766350 5520 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:33:47.766783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:47.766434 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:48.722292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:48.721188 5520 generic.go:356] "Generic (PLEG): container finished" podID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerID="8c4a7e96b51370e34d1bcb220d9c66165402fc62d665343cec970e598188ffa9" exitCode=0 Oct 31 03:33:48.722292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:48.721346 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" event={"ID":"25c3c6f6-df1a-4885-96ed-5a840a5f41c6","Type":"ContainerDied","Data":"8c4a7e96b51370e34d1bcb220d9c66165402fc62d665343cec970e598188ffa9"} Oct 31 03:33:48.722292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:48.721862 5520 scope.go:122] "RemoveContainer" containerID="8c4a7e96b51370e34d1bcb220d9c66165402fc62d665343cec970e598188ffa9" Oct 31 03:33:48.738822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:48.738782 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:33:48.739055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:48.739038 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:33:48.739186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:48.739170 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:33:49.730865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:49.730828 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" event={"ID":"25c3c6f6-df1a-4885-96ed-5a840a5f41c6","Type":"ContainerStarted","Data":"401b4295ce180f76c7a4b1786fb3b292d7f2666b064e7a4da3f89da849e74f81"} Oct 31 03:33:50.344229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:50.344006 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:50.348143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:50.348109 5520 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 03:33:50.348458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:50.348420 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 03:33:51.337148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:51.337111 5520 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 03:33:51.337641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:51.337177 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 03:33:52.338257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:52.336170 5520 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 03:33:52.338257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:52.336260 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 03:33:53.336732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:53.336698 5520 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 03:33:53.336920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:53.336753 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 03:33:54.336827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:54.336794 5520 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 03:33:54.337341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:54.336855 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 03:33:55.339651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:55.339612 5520 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 03:33:55.340322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:55.340276 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 03:33:56.340799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:56.340769 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:56.341344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:56.341322 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:56.342084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:56.342061 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:33:59.097328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.097290 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh"] Oct 31 03:33:59.101783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.101761 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.111874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.111844 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5396c2\"" Oct 31 03:33:59.114050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.114028 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh"] Oct 31 03:33:59.141434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.141402 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6"] Oct 31 03:33:59.145644 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.145620 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.150334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.150311 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6"] Oct 31 03:33:59.152051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.152030 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5396c2-dockercfg-f4vhc\"" Oct 31 03:33:59.162021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.161994 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnj7\" (UniqueName: \"kubernetes.io/projected/6388c710-6b06-4d75-a1c8-945d2c462dd3-kube-api-access-bsnj7\") pod \"payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.162143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.162091 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba243e9-16f1-45ff-9133-924a298d0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.162143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.162123 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6388c710-6b06-4d75-a1c8-945d2c462dd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.162143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.162141 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5pl\" (UniqueName: \"kubernetes.io/projected/6ba243e9-16f1-45ff-9133-924a298d0fe6-kube-api-access-5p5pl\") pod \"payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.308078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.284521 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnj7\" (UniqueName: \"kubernetes.io/projected/6388c710-6b06-4d75-a1c8-945d2c462dd3-kube-api-access-bsnj7\") pod \"payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.308078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.307452 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba243e9-16f1-45ff-9133-924a298d0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.308078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.307502 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6388c710-6b06-4d75-a1c8-945d2c462dd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.308078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.307538 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5pl\" (UniqueName: \"kubernetes.io/projected/6ba243e9-16f1-45ff-9133-924a298d0fe6-kube-api-access-5p5pl\") pod \"payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.312232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.308814 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba243e9-16f1-45ff-9133-924a298d0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.312232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.311087 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6388c710-6b06-4d75-a1c8-945d2c462dd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.332283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.332249 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnj7\" (UniqueName: \"kubernetes.io/projected/6388c710-6b06-4d75-a1c8-945d2c462dd3-kube-api-access-bsnj7\") pod \"payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.333163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.333084 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5pl\" (UniqueName: \"kubernetes.io/projected/6ba243e9-16f1-45ff-9133-924a298d0fe6-kube-api-access-5p5pl\") pod \"payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.432280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.427490 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:33:59.465517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.465481 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:33:59.835606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.835575 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh"] Oct 31 03:33:59.835606 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:59.835592 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6388c710_6b06_4d75_a1c8_945d2c462dd3.slice/crio-13bf1075e52a6070a918f5810a46364c097c5d86c062a9020383a008d2cd1abc WatchSource:0}: Error finding container 13bf1075e52a6070a918f5810a46364c097c5d86c062a9020383a008d2cd1abc: Status 404 returned error can't find the container with id 13bf1075e52a6070a918f5810a46364c097c5d86c062a9020383a008d2cd1abc Oct 31 03:33:59.851997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.851971 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6"] Oct 31 03:33:59.852589 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:33:59.852503 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba243e9_16f1_45ff_9133_924a298d0fe6.slice/crio-7e9873b2986e576420fb88f96967b7aac39741cfc2dfbf507601117410eac50a WatchSource:0}: Error finding container 7e9873b2986e576420fb88f96967b7aac39741cfc2dfbf507601117410eac50a: Status 404 returned error can't find the container with id 7e9873b2986e576420fb88f96967b7aac39741cfc2dfbf507601117410eac50a Oct 31 03:33:59.924260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.924234 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" event={"ID":"6388c710-6b06-4d75-a1c8-945d2c462dd3","Type":"ContainerStarted","Data":"13bf1075e52a6070a918f5810a46364c097c5d86c062a9020383a008d2cd1abc"} Oct 31 03:33:59.925580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:33:59.925553 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" event={"ID":"6ba243e9-16f1-45ff-9133-924a298d0fe6","Type":"ContainerStarted","Data":"7e9873b2986e576420fb88f96967b7aac39741cfc2dfbf507601117410eac50a"} Oct 31 03:34:00.934528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:00.934491 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" event={"ID":"6388c710-6b06-4d75-a1c8-945d2c462dd3","Type":"ContainerStarted","Data":"bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c"} Oct 31 03:34:00.935028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:00.934555 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:34:00.936457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:00.936430 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" event={"ID":"6ba243e9-16f1-45ff-9133-924a298d0fe6","Type":"ContainerStarted","Data":"b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec"} Oct 31 03:34:00.936597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:00.936560 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:34:00.953175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:00.953134 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" podStartSLOduration=1.953120199 podStartE2EDuration="1.953120199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:00.950725208 +0000 UTC m=+1065.145248328" watchObservedRunningTime="2026-10-31 03:34:00.953120199 +0000 UTC m=+1065.147643354" Oct 31 03:34:00.976728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:00.976674 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" podStartSLOduration=1.976657535 podStartE2EDuration="1.976657535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:00.969416388 +0000 UTC m=+1065.163939510" watchObservedRunningTime="2026-10-31 03:34:00.976657535 +0000 UTC m=+1065.171180658" Oct 31 03:34:03.789556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.789515 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd"] Oct 31 03:34:03.802730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.802698 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd"] Oct 31 03:34:03.812513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.812258 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:03.819512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.819484 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5396c2-serving-cert\"" Oct 31 03:34:03.927280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.927244 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxc66\" (UniqueName: \"kubernetes.io/projected/462c66e6-49cc-4676-b108-e10daa5b4fc5-kube-api-access-nxc66\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:03.927449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.927412 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:03.927575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.927558 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:03.927653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:03.927595 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c66e6-49cc-4676-b108-e10daa5b4fc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.036225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.032535 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxc66\" (UniqueName: \"kubernetes.io/projected/462c66e6-49cc-4676-b108-e10daa5b4fc5-kube-api-access-nxc66\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.036225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.033049 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.036225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.033269 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.036225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.033326 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c66e6-49cc-4676-b108-e10daa5b4fc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.063276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.058680 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.063276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.061002 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c66e6-49cc-4676-b108-e10daa5b4fc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.102172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.069956 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxc66\" (UniqueName: \"kubernetes.io/projected/462c66e6-49cc-4676-b108-e10daa5b4fc5-kube-api-access-nxc66\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.139223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.136305 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:04.448446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:04.448411 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:05.020915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:05.020885 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd"] Oct 31 03:34:05.021830 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:34:05.021804 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462c66e6_49cc_4676_b108_e10daa5b4fc5.slice/crio-5de86b185de6ad84d7913fb696a3f4c7cbb6ab186f9909e87fa50516dc7d11f7 WatchSource:0}: Error finding container 5de86b185de6ad84d7913fb696a3f4c7cbb6ab186f9909e87fa50516dc7d11f7: Status 404 returned error can't find the container with id 5de86b185de6ad84d7913fb696a3f4c7cbb6ab186f9909e87fa50516dc7d11f7 Oct 31 03:34:06.052758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:06.052718 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" event={"ID":"462c66e6-49cc-4676-b108-e10daa5b4fc5","Type":"ContainerStarted","Data":"605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3"} Oct 31 03:34:06.053463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:06.053356 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" event={"ID":"462c66e6-49cc-4676-b108-e10daa5b4fc5","Type":"ContainerStarted","Data":"5de86b185de6ad84d7913fb696a3f4c7cbb6ab186f9909e87fa50516dc7d11f7"} Oct 31 03:34:06.054299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:06.054269 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:09.360222 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:34:09.360148 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:34:11.873230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:11.868323 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:11.974502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:11.974470 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:34:11.978394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:11.978370 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:34:12.126290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:12.093866 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" podStartSLOduration=8.567401711 podStartE2EDuration="9.093844465s" totalImagesPullingTime="526.442754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:06.091410751 +0000 UTC m=+1070.285933890" watchObservedRunningTime="2026-10-31 03:34:12.093844465 +0000 UTC m=+1076.288367587" Oct 31 03:34:12.167230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:12.097516 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:34:12.308582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:12.098382 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:34:12.808228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:12.769577 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-77vz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:34:12.808228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:12.769667 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:12.860230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:12.838105 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.056229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.044832 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.078669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.078573 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.128424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.128371 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.268249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.267163 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:34:13.328265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.301233 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:34:13.358284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.302264 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:34:13.358284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.313300 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.358284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.357427 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:34:13.812229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.811703 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.816227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.813801 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.901229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.900302 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.901229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.900914 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:34:13.962370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:13.962324 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.131127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.075158 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.131127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.080925 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.131127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.081240 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded" Oct 31 03:34:14.184724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.183701 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.200824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.200793 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:34:14.201125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.201109 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:34:14.201302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.201289 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:34:14.300765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.300720 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.345405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.345377 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:34:14.383277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.379587 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.462238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.435977 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:34:14.801336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.801302 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:34:14.824228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.814184 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.824450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.822312 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:34:14.825050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.825028 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:34:14.911227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.910335 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.944947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.944921 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:34:14.974257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.970268 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:34:14.974257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:14.970812 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:34:15.044078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:15.038098 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:34:15.049538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:15.047911 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:34:15.052275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:15.052179 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:34:15.106221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:15.096165 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:34:18.952698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:18.952664 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7267-0a2304"] Oct 31 03:34:18.957016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:18.956987 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:18.964428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:18.964404 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xd6kv\"" Oct 31 03:34:18.969386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:18.969357 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-0a2304"] Oct 31 03:34:19.138448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:19.138411 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8gk\" (UniqueName: \"kubernetes.io/projected/61f0de80-2663-45aa-9e26-9da164a10417-kube-api-access-mb8gk\") pod \"test-curl-7267-0a2304\" (UID: \"61f0de80-2663-45aa-9e26-9da164a10417\") " pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:19.240221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:19.240114 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8gk\" (UniqueName: \"kubernetes.io/projected/61f0de80-2663-45aa-9e26-9da164a10417-kube-api-access-mb8gk\") pod \"test-curl-7267-0a2304\" (UID: \"61f0de80-2663-45aa-9e26-9da164a10417\") " pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:19.248071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:19.248045 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8gk\" (UniqueName: \"kubernetes.io/projected/61f0de80-2663-45aa-9e26-9da164a10417-kube-api-access-mb8gk\") pod \"test-curl-7267-0a2304\" (UID: \"61f0de80-2663-45aa-9e26-9da164a10417\") " pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:19.281216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:19.281176 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:19.676592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:19.676561 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-0a2304"] Oct 31 03:34:19.677049 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:34:19.677014 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61f0de80_2663_45aa_9e26_9da164a10417.slice/crio-29f24d9293c2fd2970d9c0975b8db051e657dbe8d6a72850f135a3f7706de1f9 WatchSource:0}: Error finding container 29f24d9293c2fd2970d9c0975b8db051e657dbe8d6a72850f135a3f7706de1f9: Status 404 returned error can't find the container with id 29f24d9293c2fd2970d9c0975b8db051e657dbe8d6a72850f135a3f7706de1f9 Oct 31 03:34:20.670416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:20.670298 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-0a2304" event={"ID":"61f0de80-2663-45aa-9e26-9da164a10417","Type":"ContainerStarted","Data":"29f24d9293c2fd2970d9c0975b8db051e657dbe8d6a72850f135a3f7706de1f9"} Oct 31 03:34:23.739101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:23.739064 5520 generic.go:356] "Generic (PLEG): container finished" podID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerID="4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1" exitCode=0 Oct 31 03:34:23.739501 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:23.739138 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" event={"ID":"3a1d47d5-5821-40d5-8a86-ef63a1148ae6","Type":"ContainerDied","Data":"4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1"} Oct 31 03:34:23.739745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:23.739728 5520 scope.go:122] "RemoveContainer" containerID="4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1" Oct 31 03:34:23.741855 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:23.741821 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:34:23.741994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:23.741908 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:34:23.741994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:23.741961 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:34:24.748293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:24.748242 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" event={"ID":"3a1d47d5-5821-40d5-8a86-ef63a1148ae6","Type":"ContainerStarted","Data":"dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10"} Oct 31 03:34:24.750153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:24.750121 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-0a2304" event={"ID":"61f0de80-2663-45aa-9e26-9da164a10417","Type":"ContainerStarted","Data":"15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4"} Oct 31 03:34:24.898941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:24.893094 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7267-0a2304" podStartSLOduration=3.056941888 podStartE2EDuration="6.89307333s" totalImagesPullingTime="3.836131442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:24.789731366 +0000 UTC m=+1088.984254489" watchObservedRunningTime="2026-10-31 03:34:24.89307333 +0000 UTC m=+1089.087596453" Oct 31 03:34:25.521285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.508250 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7267-0a2304"] Oct 31 03:34:25.567262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.565271 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7267-0a2304"] Oct 31 03:34:25.674134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.667275 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7267-06eeca"] Oct 31 03:34:25.675484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.675456 5520 cpu_manager.go:414] "containerMap: removing container" podUID="61f0de80-2663-45aa-9e26-9da164a10417" containerName="test-curl-7267-0a2304" Oct 31 03:34:25.675658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.675644 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f0de80-2663-45aa-9e26-9da164a10417" containerName="test-curl-7267-0a2304" Oct 31 03:34:25.675945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.675932 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f0de80-2663-45aa-9e26-9da164a10417" containerName="test-curl-7267-0a2304" Oct 31 03:34:25.702170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.702136 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 03:34:25.702504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.702457 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 03:34:25.705478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.705443 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:34:25.705601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.705558 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:25.707744 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.707722 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-06eeca"] Oct 31 03:34:25.796286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.795643 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48g4t\" (UniqueName: \"kubernetes.io/projected/ab2076ec-8664-4cca-8862-91db7c357d58-kube-api-access-48g4t\") pod \"test-curl-7267-06eeca\" (UID: \"ab2076ec-8664-4cca-8862-91db7c357d58\") " pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:25.899264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.899190 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48g4t\" (UniqueName: \"kubernetes.io/projected/ab2076ec-8664-4cca-8862-91db7c357d58-kube-api-access-48g4t\") pod \"test-curl-7267-06eeca\" (UID: \"ab2076ec-8664-4cca-8862-91db7c357d58\") " pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:25.920903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:25.920863 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48g4t\" (UniqueName: \"kubernetes.io/projected/ab2076ec-8664-4cca-8862-91db7c357d58-kube-api-access-48g4t\") pod \"test-curl-7267-06eeca\" (UID: \"ab2076ec-8664-4cca-8862-91db7c357d58\") " pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:26.031833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.030450 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:26.537729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.537694 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-06eeca"] Oct 31 03:34:26.698448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.698411 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 03:34:26.698623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.698471 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 03:34:26.822224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.821254 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7267-0a2304" podUID="61f0de80-2663-45aa-9e26-9da164a10417" containerName="test-curl-7267-0a2304" containerID="cri-o://15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4" gracePeriod=2 Oct 31 03:34:26.822743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.822458 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-06eeca" event={"ID":"ab2076ec-8664-4cca-8862-91db7c357d58","Type":"ContainerStarted","Data":"34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397"} Oct 31 03:34:26.822743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.822503 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-06eeca" event={"ID":"ab2076ec-8664-4cca-8862-91db7c357d58","Type":"ContainerStarted","Data":"93d8f61e96a225040519a95b765d66bf438f277a9df21d013b91a525af885672"} Oct 31 03:34:26.853424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:26.853368 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7267-06eeca" podStartSLOduration=1.853350882 podStartE2EDuration="1.853350882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:26.849265961 +0000 UTC m=+1091.043789082" watchObservedRunningTime="2026-10-31 03:34:26.853350882 +0000 UTC m=+1091.047874006" Oct 31 03:34:27.234191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.234154 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:34:27.698574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.698532 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 03:34:27.698754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.698604 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 03:34:27.718965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.718929 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7267-06eeca"] Oct 31 03:34:27.729860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.729823 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7267-06eeca"] Oct 31 03:34:27.898366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.898318 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7267-c01a17"] Oct 31 03:34:27.899542 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.899514 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ab2076ec-8664-4cca-8862-91db7c357d58" containerName="test-curl-7267-06eeca" Oct 31 03:34:27.899542 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.899542 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2076ec-8664-4cca-8862-91db7c357d58" containerName="test-curl-7267-06eeca" Oct 31 03:34:27.899752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.899738 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2076ec-8664-4cca-8862-91db7c357d58" containerName="test-curl-7267-06eeca" Oct 31 03:34:27.922735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.922700 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-c01a17"] Oct 31 03:34:27.923086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:27.922710 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-c01a17" Oct 31 03:34:28.051286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.048912 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wllcd\" (UniqueName: \"kubernetes.io/projected/33bb813c-1789-410b-a2cc-bf01ece9ad1a-kube-api-access-wllcd\") pod \"test-curl-7267-c01a17\" (UID: \"33bb813c-1789-410b-a2cc-bf01ece9ad1a\") " pod="opendatahub/test-curl-7267-c01a17" Oct 31 03:34:28.210381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.209706 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wllcd\" (UniqueName: \"kubernetes.io/projected/33bb813c-1789-410b-a2cc-bf01ece9ad1a-kube-api-access-wllcd\") pod \"test-curl-7267-c01a17\" (UID: \"33bb813c-1789-410b-a2cc-bf01ece9ad1a\") " pod="opendatahub/test-curl-7267-c01a17" Oct 31 03:34:28.219416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.219383 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wllcd\" (UniqueName: \"kubernetes.io/projected/33bb813c-1789-410b-a2cc-bf01ece9ad1a-kube-api-access-wllcd\") pod \"test-curl-7267-c01a17\" (UID: \"33bb813c-1789-410b-a2cc-bf01ece9ad1a\") " pod="opendatahub/test-curl-7267-c01a17" Oct 31 03:34:28.244115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.244085 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-c01a17" Oct 31 03:34:28.552990 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:34:28.552953 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33bb813c_1789_410b_a2cc_bf01ece9ad1a.slice/crio-c11a59e6075284f519761a7b3257a1451f241b7a803d9887688982e6f3df4394 WatchSource:0}: Error finding container c11a59e6075284f519761a7b3257a1451f241b7a803d9887688982e6f3df4394: Status 404 returned error can't find the container with id c11a59e6075284f519761a7b3257a1451f241b7a803d9887688982e6f3df4394 Oct 31 03:34:28.554117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.554086 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-c01a17"] Oct 31 03:34:28.713047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.704197 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 03:34:28.713047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.704314 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 03:34:28.839035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.839003 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-c01a17" event={"ID":"33bb813c-1789-410b-a2cc-bf01ece9ad1a","Type":"ContainerStarted","Data":"20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e"} Oct 31 03:34:28.839035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.839030 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7267-06eeca" podUID="ab2076ec-8664-4cca-8862-91db7c357d58" containerName="test-curl-7267-06eeca" containerID="cri-o://34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397" gracePeriod=2 Oct 31 03:34:28.839337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.839047 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-c01a17" event={"ID":"33bb813c-1789-410b-a2cc-bf01ece9ad1a","Type":"ContainerStarted","Data":"c11a59e6075284f519761a7b3257a1451f241b7a803d9887688982e6f3df4394"} Oct 31 03:34:28.854578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:28.854537 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7267-c01a17" podStartSLOduration=1.8545221710000002 podStartE2EDuration="1.854522171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:28.852907644 +0000 UTC m=+1093.047430766" watchObservedRunningTime="2026-10-31 03:34:28.854522171 +0000 UTC m=+1093.049045292" Oct 31 03:34:29.117159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.117129 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:29.255721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.253057 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f0de80-2663-45aa-9e26-9da164a10417-kube-api-access-mb8gk\" (UniqueName: \"kubernetes.io/projected/61f0de80-2663-45aa-9e26-9da164a10417-kube-api-access-mb8gk\") pod \"61f0de80-2663-45aa-9e26-9da164a10417\" (UID: \"61f0de80-2663-45aa-9e26-9da164a10417\") " Oct 31 03:34:29.277804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.265416 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f0de80-2663-45aa-9e26-9da164a10417-kube-api-access-mb8gk" pod "61f0de80-2663-45aa-9e26-9da164a10417" (UID: "61f0de80-2663-45aa-9e26-9da164a10417"). InnerVolumeSpecName "kube-api-access-mb8gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:29.361282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.361248 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb8gk\" (UniqueName: \"kubernetes.io/projected/61f0de80-2663-45aa-9e26-9da164a10417-kube-api-access-mb8gk\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:34:29.600577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.600436 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7267-c01a17"] Oct 31 03:34:29.606860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.606821 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7267-c01a17"] Oct 31 03:34:29.700715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.700664 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 03:34:29.701087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.701049 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 03:34:29.756003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.755969 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7267-a68cf2"] Oct 31 03:34:29.756673 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.756658 5520 cpu_manager.go:414] "containerMap: removing container" podUID="33bb813c-1789-410b-a2cc-bf01ece9ad1a" containerName="test-curl-7267-c01a17" Oct 31 03:34:29.756747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.756673 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33bb813c-1789-410b-a2cc-bf01ece9ad1a" containerName="test-curl-7267-c01a17" Oct 31 03:34:29.756783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.756759 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="33bb813c-1789-410b-a2cc-bf01ece9ad1a" containerName="test-curl-7267-c01a17" Oct 31 03:34:29.760739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.760714 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-a68cf2" Oct 31 03:34:29.767155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.767126 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-a68cf2"] Oct 31 03:34:29.830349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.830314 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bfn\" (UniqueName: \"kubernetes.io/projected/f21dc10e-629e-4f78-8e3d-db831d018b8c-kube-api-access-b6bfn\") pod \"test-curl-7267-a68cf2\" (UID: \"f21dc10e-629e-4f78-8e3d-db831d018b8c\") " pod="opendatahub/test-curl-7267-a68cf2" Oct 31 03:34:29.850162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.850129 5520 generic.go:356] "Generic (PLEG): container finished" podID="61f0de80-2663-45aa-9e26-9da164a10417" containerID="15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4" exitCode=137 Oct 31 03:34:29.850358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.850185 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:29.850358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.850246 5520 scope.go:122] "RemoveContainer" containerID="15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4" Oct 31 03:34:29.860934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.860915 5520 scope.go:122] "RemoveContainer" containerID="15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4" Oct 31 03:34:29.861034 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:34:29.861011 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7267-0a2304\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="61f0de80-2663-45aa-9e26-9da164a10417" pod="opendatahub/test-curl-7267-0a2304" Oct 31 03:34:29.861299 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:34:29.861270 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4\": container with ID starting with 15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4 not found: ID does not exist" containerID="15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4" Oct 31 03:34:29.861360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.861302 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4"} err="failed to get container status \"15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4\": rpc error: code = NotFound desc = could not find container \"15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4\": container with ID starting with 15f0696f700e7c0eb72fc83e1fe8049464853d0f5e2c797c6281280ad2eb87e4 not found: ID does not exist" Oct 31 03:34:29.939227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.932502 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bfn\" (UniqueName: \"kubernetes.io/projected/f21dc10e-629e-4f78-8e3d-db831d018b8c-kube-api-access-b6bfn\") pod \"test-curl-7267-a68cf2\" (UID: \"f21dc10e-629e-4f78-8e3d-db831d018b8c\") " pod="opendatahub/test-curl-7267-a68cf2" Oct 31 03:34:29.975384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:29.968822 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bfn\" (UniqueName: \"kubernetes.io/projected/f21dc10e-629e-4f78-8e3d-db831d018b8c-kube-api-access-b6bfn\") pod \"test-curl-7267-a68cf2\" (UID: \"f21dc10e-629e-4f78-8e3d-db831d018b8c\") " pod="opendatahub/test-curl-7267-a68cf2" Oct 31 03:34:30.041031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.040998 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f0de80-2663-45aa-9e26-9da164a10417" path="/var/lib/kubelet/pods/61f0de80-2663-45aa-9e26-9da164a10417/volumes" Oct 31 03:34:30.126483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.126388 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-a68cf2" Oct 31 03:34:30.429213 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.429174 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7267-a68cf2"] Oct 31 03:34:30.429777 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:34:30.429748 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf21dc10e_629e_4f78_8e3d_db831d018b8c.slice/crio-f8447704343cf249fb875d28ad4556cabb12474c54626d6cea41e26c6df9abf6 WatchSource:0}: Error finding container f8447704343cf249fb875d28ad4556cabb12474c54626d6cea41e26c6df9abf6: Status 404 returned error can't find the container with id f8447704343cf249fb875d28ad4556cabb12474c54626d6cea41e26c6df9abf6 Oct 31 03:34:30.704381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.704353 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:34:30.705596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.705572 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:34:30.879951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.879919 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-a68cf2" event={"ID":"f21dc10e-629e-4f78-8e3d-db831d018b8c","Type":"ContainerStarted","Data":"ecfa7eca74b51ef64ce85041d57574e71e9d205dbee31c764dbc74b13c8fc37f"} Oct 31 03:34:30.880099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.879969 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7267-a68cf2" event={"ID":"f21dc10e-629e-4f78-8e3d-db831d018b8c","Type":"ContainerStarted","Data":"f8447704343cf249fb875d28ad4556cabb12474c54626d6cea41e26c6df9abf6"} Oct 31 03:34:30.881152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.881122 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7267-c01a17" podUID="33bb813c-1789-410b-a2cc-bf01ece9ad1a" containerName="test-curl-7267-c01a17" containerID="cri-o://20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e" gracePeriod=2 Oct 31 03:34:30.894923 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:30.894878 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7267-a68cf2" podStartSLOduration=1.8948623470000001 podStartE2EDuration="1.894862347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:30.893152029 +0000 UTC m=+1095.087675236" watchObservedRunningTime="2026-10-31 03:34:30.894862347 +0000 UTC m=+1095.089385468" Oct 31 03:34:31.067920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.067882 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:31.075991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.075965 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2076ec-8664-4cca-8862-91db7c357d58-kube-api-access-48g4t\" (UniqueName: \"kubernetes.io/projected/ab2076ec-8664-4cca-8862-91db7c357d58-kube-api-access-48g4t\") pod \"ab2076ec-8664-4cca-8862-91db7c357d58\" (UID: \"ab2076ec-8664-4cca-8862-91db7c357d58\") " Oct 31 03:34:31.078659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.078629 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2076ec-8664-4cca-8862-91db7c357d58-kube-api-access-48g4t" pod "ab2076ec-8664-4cca-8862-91db7c357d58" (UID: "ab2076ec-8664-4cca-8862-91db7c357d58"). InnerVolumeSpecName "kube-api-access-48g4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:31.178179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.178142 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48g4t\" (UniqueName: \"kubernetes.io/projected/ab2076ec-8664-4cca-8862-91db7c357d58-kube-api-access-48g4t\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:34:31.556607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.556553 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7267-a68cf2"] Oct 31 03:34:31.560335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.560309 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7267-a68cf2"] Oct 31 03:34:31.891992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.891905 5520 generic.go:356] "Generic (PLEG): container finished" podID="ab2076ec-8664-4cca-8862-91db7c357d58" containerID="34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397" exitCode=137 Oct 31 03:34:31.891992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.891948 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:31.891992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.891989 5520 scope.go:122] "RemoveContainer" containerID="34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397" Oct 31 03:34:31.945825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.906626 5520 scope.go:122] "RemoveContainer" containerID="34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397" Oct 31 03:34:31.970247 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:34:31.965389 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397\": container with ID starting with 34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397 not found: ID does not exist" containerID="34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397" Oct 31 03:34:31.970247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:31.965438 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397"} err="failed to get container status \"34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397\": rpc error: code = NotFound desc = could not find container \"34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397\": container with ID starting with 34f466639e4e20ad224d104543f96b145357317fb201fd884da754937a9cb397 not found: ID does not exist" Oct 31 03:34:32.015227 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:34:31.978966 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7267-06eeca\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="ab2076ec-8664-4cca-8862-91db7c357d58" pod="opendatahub/test-curl-7267-06eeca" Oct 31 03:34:32.035424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:32.035379 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2076ec-8664-4cca-8862-91db7c357d58" path="/var/lib/kubelet/pods/ab2076ec-8664-4cca-8862-91db7c357d58/volumes" Oct 31 03:34:32.930452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:32.930417 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7267-a68cf2" podUID="f21dc10e-629e-4f78-8e3d-db831d018b8c" containerName="test-curl-7267-a68cf2" containerID="cri-o://ecfa7eca74b51ef64ce85041d57574e71e9d205dbee31c764dbc74b13c8fc37f" gracePeriod=2 Oct 31 03:34:33.164889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:33.164818 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-c01a17" Oct 31 03:34:33.274578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:33.274529 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bb813c-1789-410b-a2cc-bf01ece9ad1a-kube-api-access-wllcd\" (UniqueName: \"kubernetes.io/projected/33bb813c-1789-410b-a2cc-bf01ece9ad1a-kube-api-access-wllcd\") pod \"33bb813c-1789-410b-a2cc-bf01ece9ad1a\" (UID: \"33bb813c-1789-410b-a2cc-bf01ece9ad1a\") " Oct 31 03:34:33.277004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:33.276976 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bb813c-1789-410b-a2cc-bf01ece9ad1a-kube-api-access-wllcd" pod "33bb813c-1789-410b-a2cc-bf01ece9ad1a" (UID: "33bb813c-1789-410b-a2cc-bf01ece9ad1a"). InnerVolumeSpecName "kube-api-access-wllcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:33.377357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:33.377311 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wllcd\" (UniqueName: \"kubernetes.io/projected/33bb813c-1789-410b-a2cc-bf01ece9ad1a-kube-api-access-wllcd\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:34:33.973351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:33.972192 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33bb813c-1789-410b-a2cc-bf01ece9ad1a" path="/var/lib/kubelet/pods/33bb813c-1789-410b-a2cc-bf01ece9ad1a/volumes" Oct 31 03:34:34.003386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:34.003350 5520 generic.go:356] "Generic (PLEG): container finished" podID="33bb813c-1789-410b-a2cc-bf01ece9ad1a" containerID="20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e" exitCode=137 Oct 31 03:34:34.003759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:34.003503 5520 scope.go:122] "RemoveContainer" containerID="20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e" Oct 31 03:34:34.003869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:34.003604 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-c01a17" Oct 31 03:34:34.035445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:34.031052 5520 scope.go:122] "RemoveContainer" containerID="20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e" Oct 31 03:34:34.035445 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:34:34.032343 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e\": container with ID starting with 20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e not found: ID does not exist" containerID="20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e" Oct 31 03:34:34.035445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:34.032375 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e"} err="failed to get container status \"20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e\": rpc error: code = NotFound desc = could not find container \"20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e\": container with ID starting with 20b2afac4baeaa82a22c31e643c1ba26e64e6faabaaa6a3b8cd620017f91a68e not found: ID does not exist" Oct 31 03:34:35.024641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:35.022042 5520 generic.go:356] "Generic (PLEG): container finished" podID="f21dc10e-629e-4f78-8e3d-db831d018b8c" containerID="ecfa7eca74b51ef64ce85041d57574e71e9d205dbee31c764dbc74b13c8fc37f" exitCode=137 Oct 31 03:34:35.195275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:35.195246 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-a68cf2" Oct 31 03:34:35.333732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:35.333653 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21dc10e-629e-4f78-8e3d-db831d018b8c-kube-api-access-b6bfn\" (UniqueName: \"kubernetes.io/projected/f21dc10e-629e-4f78-8e3d-db831d018b8c-kube-api-access-b6bfn\") pod \"f21dc10e-629e-4f78-8e3d-db831d018b8c\" (UID: \"f21dc10e-629e-4f78-8e3d-db831d018b8c\") " Oct 31 03:34:35.336269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:35.336235 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21dc10e-629e-4f78-8e3d-db831d018b8c-kube-api-access-b6bfn" pod "f21dc10e-629e-4f78-8e3d-db831d018b8c" (UID: "f21dc10e-629e-4f78-8e3d-db831d018b8c"). InnerVolumeSpecName "kube-api-access-b6bfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:34:35.437045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:35.437010 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bfn\" (UniqueName: \"kubernetes.io/projected/f21dc10e-629e-4f78-8e3d-db831d018b8c-kube-api-access-b6bfn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:34:35.923611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:35.923572 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21dc10e-629e-4f78-8e3d-db831d018b8c" path="/var/lib/kubelet/pods/f21dc10e-629e-4f78-8e3d-db831d018b8c/volumes" Oct 31 03:34:36.033184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:36.033152 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7267-a68cf2" Oct 31 03:34:36.033706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:36.033156 5520 scope.go:122] "RemoveContainer" containerID="ecfa7eca74b51ef64ce85041d57574e71e9d205dbee31c764dbc74b13c8fc37f" Oct 31 03:34:37.199564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.199533 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp"] Oct 31 03:34:37.200236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.200219 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f21dc10e-629e-4f78-8e3d-db831d018b8c" containerName="test-curl-7267-a68cf2" Oct 31 03:34:37.200300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.200238 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21dc10e-629e-4f78-8e3d-db831d018b8c" containerName="test-curl-7267-a68cf2" Oct 31 03:34:37.200401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.200391 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21dc10e-629e-4f78-8e3d-db831d018b8c" containerName="test-curl-7267-a68cf2" Oct 31 03:34:37.205987 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.205962 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:37.281911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.261557 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-34cb76-dockercfg-qbx54\"" Oct 31 03:34:37.282428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.282377 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-34cb76\"" Oct 31 03:34:37.312251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.286765 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gg4\" (UniqueName: \"kubernetes.io/projected/28f034f9-48d8-4626-a8e0-a0ff289a3adf-kube-api-access-c9gg4\") pod \"payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:37.312251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.286989 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f034f9-48d8-4626-a8e0-a0ff289a3adf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:37.452346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.451575 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp"] Oct 31 03:34:37.465359 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.465322 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gg4\" (UniqueName: \"kubernetes.io/projected/28f034f9-48d8-4626-a8e0-a0ff289a3adf-kube-api-access-c9gg4\") pod \"payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:37.587342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.576472 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gg4\" (UniqueName: \"kubernetes.io/projected/28f034f9-48d8-4626-a8e0-a0ff289a3adf-kube-api-access-c9gg4\") pod \"payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:37.587689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.587655 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f034f9-48d8-4626-a8e0-a0ff289a3adf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:37.589453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.589426 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f034f9-48d8-4626-a8e0-a0ff289a3adf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:37.632371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:37.632340 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:38.003998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:38.002350 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp"] Oct 31 03:34:38.056366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:38.056328 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" event={"ID":"28f034f9-48d8-4626-a8e0-a0ff289a3adf","Type":"ContainerStarted","Data":"5d5d6597b1dffd397e8553376a1db8dd88e9dc52675c148658049e25ce53417f"} Oct 31 03:34:39.067846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:39.067808 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" event={"ID":"28f034f9-48d8-4626-a8e0-a0ff289a3adf","Type":"ContainerStarted","Data":"3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198"} Oct 31 03:34:39.068345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:39.068081 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:34:39.101888 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:39.101798 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" podStartSLOduration=2.10177893 podStartE2EDuration="2.10177893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:39.095753761 +0000 UTC m=+1103.290276882" watchObservedRunningTime="2026-10-31 03:34:39.10177893 +0000 UTC m=+1103.296302054" Oct 31 03:34:39.362831 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:34:39.362745 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:34:50.174399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:34:50.173455 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:35:01.848698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:01.848637 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7261-db2154"] Oct 31 03:35:01.853228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:01.853193 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:01.856452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:01.856423 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xd6kv\"" Oct 31 03:35:01.859858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:01.859831 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7261-db2154"] Oct 31 03:35:02.044293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:02.032021 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5jp\" (UniqueName: \"kubernetes.io/projected/820a468c-e8ae-42ac-bb33-8e5c49467060-kube-api-access-gf5jp\") pod \"test-curl-7261-db2154\" (UID: \"820a468c-e8ae-42ac-bb33-8e5c49467060\") " pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:02.158396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:02.157774 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5jp\" (UniqueName: \"kubernetes.io/projected/820a468c-e8ae-42ac-bb33-8e5c49467060-kube-api-access-gf5jp\") pod \"test-curl-7261-db2154\" (UID: \"820a468c-e8ae-42ac-bb33-8e5c49467060\") " pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:02.236930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:02.234798 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5jp\" (UniqueName: \"kubernetes.io/projected/820a468c-e8ae-42ac-bb33-8e5c49467060-kube-api-access-gf5jp\") pod \"test-curl-7261-db2154\" (UID: \"820a468c-e8ae-42ac-bb33-8e5c49467060\") " pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:02.477496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:02.477460 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:02.767596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:02.767446 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7261-db2154"] Oct 31 03:35:02.767998 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:02.767965 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod820a468c_e8ae_42ac_bb33_8e5c49467060.slice/crio-acf7e4532bc11dac2458f943cae2d9ec27d78928798f1cfbcc42f97bbd829dd6 WatchSource:0}: Error finding container acf7e4532bc11dac2458f943cae2d9ec27d78928798f1cfbcc42f97bbd829dd6: Status 404 returned error can't find the container with id acf7e4532bc11dac2458f943cae2d9ec27d78928798f1cfbcc42f97bbd829dd6 Oct 31 03:35:03.725257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:03.725224 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7261-db2154" event={"ID":"820a468c-e8ae-42ac-bb33-8e5c49467060","Type":"ContainerStarted","Data":"0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c"} Oct 31 03:35:03.725257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:03.725266 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7261-db2154" event={"ID":"820a468c-e8ae-42ac-bb33-8e5c49467060","Type":"ContainerStarted","Data":"acf7e4532bc11dac2458f943cae2d9ec27d78928798f1cfbcc42f97bbd829dd6"} Oct 31 03:35:03.743406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:03.743354 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7261-db2154" podStartSLOduration=2.743339912 podStartE2EDuration="2.743339912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:03.739039094 +0000 UTC m=+1127.933562265" watchObservedRunningTime="2026-10-31 03:35:03.743339912 +0000 UTC m=+1127.937863032" Oct 31 03:35:04.545745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:04.545582 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7261-db2154"] Oct 31 03:35:04.551183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:04.551155 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7261-db2154"] Oct 31 03:35:05.757804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:05.757765 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7261-db2154" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" containerName="test-curl-7261-db2154" containerID="cri-o://0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c" gracePeriod=2 Oct 31 03:35:05.759856 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:05.759822 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7261-db2154\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:05.949131 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:05.949083 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7261-db2154\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:08.055519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.055484 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:08.241964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.241931 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820a468c-e8ae-42ac-bb33-8e5c49467060-kube-api-access-gf5jp\" (UniqueName: \"kubernetes.io/projected/820a468c-e8ae-42ac-bb33-8e5c49467060-kube-api-access-gf5jp\") pod \"820a468c-e8ae-42ac-bb33-8e5c49467060\" (UID: \"820a468c-e8ae-42ac-bb33-8e5c49467060\") " Oct 31 03:35:08.244428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.244397 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820a468c-e8ae-42ac-bb33-8e5c49467060-kube-api-access-gf5jp" pod "820a468c-e8ae-42ac-bb33-8e5c49467060" (UID: "820a468c-e8ae-42ac-bb33-8e5c49467060"). InnerVolumeSpecName "kube-api-access-gf5jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:08.344693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.344598 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5jp\" (UniqueName: \"kubernetes.io/projected/820a468c-e8ae-42ac-bb33-8e5c49467060-kube-api-access-gf5jp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:08.766657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.762348 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg"] Oct 31 03:35:08.766657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.763533 5520 cpu_manager.go:414] "containerMap: removing container" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" containerName="test-curl-7261-db2154" Oct 31 03:35:08.766657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.763554 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" containerName="test-curl-7261-db2154" Oct 31 03:35:08.766657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.763700 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" containerName="test-curl-7261-db2154" Oct 31 03:35:08.768086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.768064 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.786634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.785422 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8afac225-openshift-default-dockercfg-b9r6x\"" Oct 31 03:35:08.797863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.797833 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg"] Oct 31 03:35:08.839183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.839151 5520 generic.go:356] "Generic (PLEG): container finished" podID="820a468c-e8ae-42ac-bb33-8e5c49467060" containerID="0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c" exitCode=137 Oct 31 03:35:08.839439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.839273 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:08.839439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.839336 5520 scope.go:122] "RemoveContainer" containerID="0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c" Oct 31 03:35:08.854140 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:08.854104 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7261-db2154\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:08.858950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.858924 5520 scope.go:122] "RemoveContainer" containerID="0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c" Oct 31 03:35:08.860383 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:08.860350 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c\": container with ID starting with 0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c not found: ID does not exist" containerID="0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c" Oct 31 03:35:08.860551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.860526 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c"} err="failed to get container status \"0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c\": rpc error: code = NotFound desc = could not find container \"0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c\": container with ID starting with 0e0a8bc03acee0e4283c0e38c15373b9a0605952578d682c91ac27875828d90c not found: ID does not exist" Oct 31 03:35:08.861260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861220 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-podinfo\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861259 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-data\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861289 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-envoy\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861339 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istiod-ca-cert\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861369 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-socket\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861464 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-credential-socket\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861497 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-token\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861578 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz7qv\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-kube-api-access-vz7qv\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.861735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.861706 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-certs\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968464 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-certs\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968578 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-podinfo\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968610 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-data\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968641 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-envoy\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968705 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istiod-ca-cert\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968736 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-socket\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968861 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-credential-socket\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.968896 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-token\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.969001 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz7qv\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-kube-api-access-vz7qv\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.981543 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-socket\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.982010 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-credential-socket\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.982158 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istiod-ca-cert\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.982445 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-data\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:08.988474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:08.982931 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-certs\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:09.046273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.045959 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-token\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:09.059268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.054898 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz7qv\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-kube-api-access-vz7qv\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:09.059268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.055474 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-podinfo\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:09.075224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.069528 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-envoy\") pod \"e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:09.101245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.098518 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:09.508855 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:09.507266 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:35:09.686313 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:09.685865 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1be6ef9_0699_4d3c_be31_9e93ace5bbe7.slice/crio-00b571c19f6db93ca0bcd0e7a98215daba083a0d5ace170d0c4c0b8e6bf4aad6 WatchSource:0}: Error finding container 00b571c19f6db93ca0bcd0e7a98215daba083a0d5ace170d0c4c0b8e6bf4aad6: Status 404 returned error can't find the container with id 00b571c19f6db93ca0bcd0e7a98215daba083a0d5ace170d0c4c0b8e6bf4aad6 Oct 31 03:35:09.686313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.686188 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg"] Oct 31 03:35:09.692448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.691775 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:35:09.692448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.691860 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:35:09.692448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.691905 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:35:09.850268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.850118 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" event={"ID":"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7","Type":"ContainerStarted","Data":"84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9"} Oct 31 03:35:09.850268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.850191 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" event={"ID":"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7","Type":"ContainerStarted","Data":"00b571c19f6db93ca0bcd0e7a98215daba083a0d5ace170d0c4c0b8e6bf4aad6"} Oct 31 03:35:09.852440 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:09.852406 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7261-db2154\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" pod="opendatahub/test-curl-7261-db2154" Oct 31 03:35:09.925249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:09.925189 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820a468c-e8ae-42ac-bb33-8e5c49467060" path="/var/lib/kubelet/pods/820a468c-e8ae-42ac-bb33-8e5c49467060/volumes" Oct 31 03:35:10.108042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:10.102333 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:10.108042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:10.103467 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:35:10.108042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:10.103522 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:35:11.158228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:11.156027 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:11.209228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:11.207111 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podStartSLOduration=3.207079912 podStartE2EDuration="3.207079912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:09.877901049 +0000 UTC m=+1134.072424198" watchObservedRunningTime="2026-10-31 03:35:11.207079912 +0000 UTC m=+1135.401603034" Oct 31 03:35:11.866448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:11.866412 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:11.867353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:11.867329 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:35:12.074728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.056903 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9"] Oct 31 03:35:12.074728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.063054 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.081160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.080820 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9"] Oct 31 03:35:12.135429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.135344 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-ab21e4\"" Oct 31 03:35:12.135429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.135344 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-ab21e4-dockercfg-cfsfw\"" Oct 31 03:35:12.202230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.202160 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.202686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.202272 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfhx4\" (UniqueName: \"kubernetes.io/projected/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-kube-api-access-sfhx4\") pod \"payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.307229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.307099 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.307439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.307244 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfhx4\" (UniqueName: \"kubernetes.io/projected/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-kube-api-access-sfhx4\") pod \"payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.321225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.319086 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.363223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.362948 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfhx4\" (UniqueName: \"kubernetes.io/projected/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-kube-api-access-sfhx4\") pod \"payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.421276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.410023 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:12.765613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.765587 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9"] Oct 31 03:35:12.767569 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:12.767538 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc45b5c9_db98_4f39_93ca_18fdf3b15db9.slice/crio-a011a7cf4b7b7f1db396de9eca838964ff541c6f0f1acf4692f7df61974ca92d WatchSource:0}: Error finding container a011a7cf4b7b7f1db396de9eca838964ff541c6f0f1acf4692f7df61974ca92d: Status 404 returned error can't find the container with id a011a7cf4b7b7f1db396de9eca838964ff541c6f0f1acf4692f7df61974ca92d Oct 31 03:35:12.877250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.877212 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" event={"ID":"bc45b5c9-db98-4f39-93ca-18fdf3b15db9","Type":"ContainerStarted","Data":"b93e9b6e343de1f1b1d0a7e3de512664410f574b65880bd768a4596cc4a8d7fa"} Oct 31 03:35:12.877421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:12.877280 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" event={"ID":"bc45b5c9-db98-4f39-93ca-18fdf3b15db9","Type":"ContainerStarted","Data":"a011a7cf4b7b7f1db396de9eca838964ff541c6f0f1acf4692f7df61974ca92d"} Oct 31 03:35:13.618363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.618314 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" podStartSLOduration=1.6182990560000001 podStartE2EDuration="1.618299056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:12.89535243 +0000 UTC m=+1137.089875552" watchObservedRunningTime="2026-10-31 03:35:13.618299056 +0000 UTC m=+1137.812822176" Oct 31 03:35:13.618782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.618669 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47"] Oct 31 03:35:13.623344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.623324 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:13.628126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.628105 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8afac225\"" Oct 31 03:35:13.628305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.628162 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8afac225-dockercfg-bmb2n\"" Oct 31 03:35:13.632671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.632645 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47"] Oct 31 03:35:13.657223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.650549 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2"] Oct 31 03:35:13.685078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.684841 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2"] Oct 31 03:35:13.685078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.684961 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:13.738995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.738960 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337a83c-9532-4771-b918-65770e4fa1cc-plugins-config-volume\") pod \"payload-processing-e2e-mt-8afac225-cd74654c9-2fc47\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:13.739229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.739034 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad527969-608c-4bdb-afe8-a4fb138623b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:13.739336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.739308 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdsht\" (UniqueName: \"kubernetes.io/projected/ad527969-608c-4bdb-afe8-a4fb138623b3-kube-api-access-zdsht\") pod \"payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:13.739544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.739523 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9t2\" (UniqueName: \"kubernetes.io/projected/5337a83c-9532-4771-b918-65770e4fa1cc-kube-api-access-6c9t2\") pod \"payload-processing-e2e-mt-8afac225-cd74654c9-2fc47\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:13.967276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.943261 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337a83c-9532-4771-b918-65770e4fa1cc-plugins-config-volume\") pod \"payload-processing-e2e-mt-8afac225-cd74654c9-2fc47\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:13.967276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.943355 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad527969-608c-4bdb-afe8-a4fb138623b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:13.967276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.943730 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdsht\" (UniqueName: \"kubernetes.io/projected/ad527969-608c-4bdb-afe8-a4fb138623b3-kube-api-access-zdsht\") pod \"payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:13.967276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.943904 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9t2\" (UniqueName: \"kubernetes.io/projected/5337a83c-9532-4771-b918-65770e4fa1cc-kube-api-access-6c9t2\") pod \"payload-processing-e2e-mt-8afac225-cd74654c9-2fc47\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:13.967276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:13.950630 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337a83c-9532-4771-b918-65770e4fa1cc-plugins-config-volume\") pod \"payload-processing-e2e-mt-8afac225-cd74654c9-2fc47\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:14.054230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.053927 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad527969-608c-4bdb-afe8-a4fb138623b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:14.211882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.211835 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9t2\" (UniqueName: \"kubernetes.io/projected/5337a83c-9532-4771-b918-65770e4fa1cc-kube-api-access-6c9t2\") pod \"payload-processing-e2e-mt-8afac225-cd74654c9-2fc47\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:14.218907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.218828 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdsht\" (UniqueName: \"kubernetes.io/projected/ad527969-608c-4bdb-afe8-a4fb138623b3-kube-api-access-zdsht\") pod \"payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:14.229749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.229722 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:14.239973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.239952 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:14.316807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.315501 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:14.624380 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:14.624345 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5337a83c_9532_4771_b918_65770e4fa1cc.slice/crio-4479fb67223cabff5b572f77657a7dde0f1db644d6a1af86fb42ea4c86b52cd2 WatchSource:0}: Error finding container 4479fb67223cabff5b572f77657a7dde0f1db644d6a1af86fb42ea4c86b52cd2: Status 404 returned error can't find the container with id 4479fb67223cabff5b572f77657a7dde0f1db644d6a1af86fb42ea4c86b52cd2 Oct 31 03:35:14.624782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.624759 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47"] Oct 31 03:35:14.735879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:14.735846 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2"] Oct 31 03:35:14.736003 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:14.735915 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad527969_608c_4bdb_afe8_a4fb138623b3.slice/crio-e35965297336c0737f878f34308805432174bcc9cc8266ccb2d2d9e4e7f4a387 WatchSource:0}: Error finding container e35965297336c0737f878f34308805432174bcc9cc8266ccb2d2d9e4e7f4a387: Status 404 returned error can't find the container with id e35965297336c0737f878f34308805432174bcc9cc8266ccb2d2d9e4e7f4a387 Oct 31 03:35:15.072267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.067761 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" event={"ID":"5337a83c-9532-4771-b918-65770e4fa1cc","Type":"ContainerStarted","Data":"cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a"} Oct 31 03:35:15.072267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.067814 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" event={"ID":"5337a83c-9532-4771-b918-65770e4fa1cc","Type":"ContainerStarted","Data":"4479fb67223cabff5b572f77657a7dde0f1db644d6a1af86fb42ea4c86b52cd2"} Oct 31 03:35:15.072267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.067950 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:15.072267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.070671 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" event={"ID":"ad527969-608c-4bdb-afe8-a4fb138623b3","Type":"ContainerStarted","Data":"93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714"} Oct 31 03:35:15.072267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.070718 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" event={"ID":"ad527969-608c-4bdb-afe8-a4fb138623b3","Type":"ContainerStarted","Data":"e35965297336c0737f878f34308805432174bcc9cc8266ccb2d2d9e4e7f4a387"} Oct 31 03:35:15.072267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.070933 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:15.084688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.084633 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" podStartSLOduration=2.084617457 podStartE2EDuration="2.084617457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:15.083425405 +0000 UTC m=+1139.277948539" watchObservedRunningTime="2026-10-31 03:35:15.084617457 +0000 UTC m=+1139.279140578" Oct 31 03:35:15.104624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.104571 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" podStartSLOduration=2.104555061 podStartE2EDuration="2.104555061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:15.098525102 +0000 UTC m=+1139.293048224" watchObservedRunningTime="2026-10-31 03:35:15.104555061 +0000 UTC m=+1139.299078182" Oct 31 03:35:15.237228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.235758 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4"] Oct 31 03:35:15.266806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:15.266768 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj"] Oct 31 03:35:17.023073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:17.023038 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj"] Oct 31 03:35:17.026728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:17.026704 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7b20be-7ktlj"] Oct 31 03:35:17.999935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:17.999878 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd" path="/var/lib/kubelet/pods/3abc4cd2-1bc2-4cf6-b344-ba4d2fb277bd/volumes" Oct 31 03:35:20.237728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:20.237661 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" podUID="da2b4e6d-7217-43b7-8154-9a417b58a292" containerName="payload-processing" containerID="cri-o://5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c" gracePeriod=25 Oct 31 03:35:20.325231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:20.318575 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" podUID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" containerName="payload-pre-processing" containerID="cri-o://9af914b16a13e41f65d4cc70119a3e41e81cae92ad7b65faee0cb5baa0ee2d3e" gracePeriod=25 Oct 31 03:35:21.509229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.507911 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk"] Oct 31 03:35:21.525675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.525641 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk"] Oct 31 03:35:21.525914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.525882 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.531828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.531396 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8afac225-serving-cert\"" Oct 31 03:35:21.599001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.598963 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-metrics-certs\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.599265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.599110 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12490f36-1e53-40e8-b0bc-57c1995ba5df-openshift-service-ca\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.599265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.599149 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvpk\" (UniqueName: \"kubernetes.io/projected/12490f36-1e53-40e8-b0bc-57c1995ba5df-kube-api-access-xjvpk\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.599265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.599252 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-maas-api-tls\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.701021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.700977 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-metrics-certs\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.701230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.701107 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12490f36-1e53-40e8-b0bc-57c1995ba5df-openshift-service-ca\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.701230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.701153 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvpk\" (UniqueName: \"kubernetes.io/projected/12490f36-1e53-40e8-b0bc-57c1995ba5df-kube-api-access-xjvpk\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.701355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.701270 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-maas-api-tls\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.702147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.702115 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12490f36-1e53-40e8-b0bc-57c1995ba5df-openshift-service-ca\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.704054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.704029 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-metrics-certs\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.704415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.704393 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-maas-api-tls\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.709818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.709789 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvpk\" (UniqueName: \"kubernetes.io/projected/12490f36-1e53-40e8-b0bc-57c1995ba5df-kube-api-access-xjvpk\") pod \"maas-api-e2e-mt-8afac225-68647fb488-7qfdk\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:21.852811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:21.851494 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:22.459729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:22.459693 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk"] Oct 31 03:35:22.463059 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:22.463011 5520 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:35:23.510769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:23.507661 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" event={"ID":"12490f36-1e53-40e8-b0bc-57c1995ba5df","Type":"ContainerStarted","Data":"2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f"} Oct 31 03:35:23.510769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:23.507732 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" event={"ID":"12490f36-1e53-40e8-b0bc-57c1995ba5df","Type":"ContainerStarted","Data":"0227262aa67f09d44b9770f0175558b76e5931b72251461488d23533534e5330"} Oct 31 03:35:23.510769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:23.508923 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:23.531717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:23.531655 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" podStartSLOduration=2.080152014 podStartE2EDuration="2.531635824s" totalImagesPullingTime="451.48381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:23.528234716 +0000 UTC m=+1147.722757864" watchObservedRunningTime="2026-10-31 03:35:23.531635824 +0000 UTC m=+1147.726158947" Oct 31 03:35:25.119228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.114376 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 03:35:25.472942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.472908 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s"] Oct 31 03:35:25.477312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.477287 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:25.501555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.501529 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d4145b-dockercfg-hxd8j\"" Oct 31 03:35:25.502157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.502131 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-d4145b\"" Oct 31 03:35:25.509438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.509412 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s"] Oct 31 03:35:25.530733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.530698 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph24p\" (UniqueName: \"kubernetes.io/projected/a4b73f30-4c6f-4f53-a679-84694df92392-kube-api-access-ph24p\") pod \"payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:25.530925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.530867 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b73f30-4c6f-4f53-a679-84694df92392-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:25.652240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.649342 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph24p\" (UniqueName: \"kubernetes.io/projected/a4b73f30-4c6f-4f53-a679-84694df92392-kube-api-access-ph24p\") pod \"payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:25.652240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.649511 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b73f30-4c6f-4f53-a679-84694df92392-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:25.652240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.650518 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b73f30-4c6f-4f53-a679-84694df92392-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:25.708230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.704654 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph24p\" (UniqueName: \"kubernetes.io/projected/a4b73f30-4c6f-4f53-a679-84694df92392-kube-api-access-ph24p\") pod \"payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:25.809293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:25.809179 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:26.093374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:26.093288 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:35:26.330666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:26.256897 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s"] Oct 31 03:35:26.536852 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:26.536813 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" event={"ID":"a4b73f30-4c6f-4f53-a679-84694df92392","Type":"ContainerStarted","Data":"a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237"} Oct 31 03:35:26.537028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:26.536882 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" event={"ID":"a4b73f30-4c6f-4f53-a679-84694df92392","Type":"ContainerStarted","Data":"41422419aea822d37f7441ff3361e4f4e2d69ca537fd8377ba4d5ed0a8187b0b"} Oct 31 03:35:26.537028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:26.536962 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:26.571410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:26.571308 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" podStartSLOduration=1.571287071 podStartE2EDuration="1.571287071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:26.568677643 +0000 UTC m=+1150.763200765" watchObservedRunningTime="2026-10-31 03:35:26.571287071 +0000 UTC m=+1150.765810194" Oct 31 03:35:27.103243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:27.103190 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:35:27.777085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:27.777049 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" podUID="da2b4e6d-7217-43b7-8154-9a417b58a292" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:35:27.778455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:27.778428 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" podUID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:35:30.539512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:30.539474 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:35:32.344934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:32.344897 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd"] Oct 31 03:35:32.345772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:32.345744 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" podUID="97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" containerName="maas-api" containerID="cri-o://594465c4821bd6fc4f740189825d8d14813f1f1430ba4cc4dec6960b59397c13" gracePeriod=30 Oct 31 03:35:32.790546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:32.790508 5520 generic.go:356] "Generic (PLEG): container finished" podID="97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" containerID="594465c4821bd6fc4f740189825d8d14813f1f1430ba4cc4dec6960b59397c13" exitCode=0 Oct 31 03:35:32.790925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:32.790808 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" event={"ID":"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6","Type":"ContainerDied","Data":"594465c4821bd6fc4f740189825d8d14813f1f1430ba4cc4dec6960b59397c13"} Oct 31 03:35:33.294945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.294916 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:35:33.377479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.377384 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-openshift-service-ca\") pod \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " Oct 31 03:35:33.377479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.377470 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-kube-api-access-zvzts\" (UniqueName: \"kubernetes.io/projected/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-kube-api-access-zvzts\") pod \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " Oct 31 03:35:33.378011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.377525 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-metrics-certs\") pod \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " Oct 31 03:35:33.378011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.377613 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls\") pod \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\" (UID: \"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6\") " Oct 31 03:35:33.378011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.377894 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-openshift-service-ca" pod "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" (UID: "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:33.378907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.378582 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:33.384480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.383461 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls" pod "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" (UID: "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:33.384480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.383716 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-kube-api-access-zvzts" pod "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" (UID: "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6"). InnerVolumeSpecName "kube-api-access-zvzts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:33.386947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.386918 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-metrics-certs" pod "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" (UID: "97bd4c3a-aaf9-4b7f-ab96-7017555a32e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:35:33.479824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.479789 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvzts\" (UniqueName: \"kubernetes.io/projected/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-kube-api-access-zvzts\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:33.479824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.479821 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:33.479824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.479831 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:33.867298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.862911 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" event={"ID":"97bd4c3a-aaf9-4b7f-ab96-7017555a32e6","Type":"ContainerDied","Data":"929da85ee08c00c13029b7a7573eedc21715eee8cb0d3a56c9625a8df11eaab5"} Oct 31 03:35:33.867298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.862991 5520 scope.go:122] "RemoveContainer" containerID="594465c4821bd6fc4f740189825d8d14813f1f1430ba4cc4dec6960b59397c13" Oct 31 03:35:33.867298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.863196 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd" Oct 31 03:35:33.941016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.940983 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd"] Oct 31 03:35:33.941016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:33.941017 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-wtgwd"] Oct 31 03:35:35.329958 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.329917 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx"] Oct 31 03:35:35.331121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.331090 5520 cpu_manager.go:414] "containerMap: removing container" podUID="97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" containerName="maas-api" Oct 31 03:35:35.331176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.331125 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" containerName="maas-api" Oct 31 03:35:35.331385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.331370 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" containerName="maas-api" Oct 31 03:35:35.337275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.337249 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx"] Oct 31 03:35:35.337398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.337355 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.343352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.343326 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-d4145b-serving-cert\"" Oct 31 03:35:35.444889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.444856 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.445068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.445041 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.445145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.445099 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4hf\" (UniqueName: \"kubernetes.io/projected/70694229-8fa5-4e3f-af7f-f875faba310b-kube-api-access-sk4hf\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.445275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.445251 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70694229-8fa5-4e3f-af7f-f875faba310b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.552232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.551748 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.552232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.551912 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.552232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.551963 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4hf\" (UniqueName: \"kubernetes.io/projected/70694229-8fa5-4e3f-af7f-f875faba310b-kube-api-access-sk4hf\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.552232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.552001 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70694229-8fa5-4e3f-af7f-f875faba310b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.559386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.553415 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70694229-8fa5-4e3f-af7f-f875faba310b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.583431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.583353 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.611228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.607388 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.611228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.608252 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4hf\" (UniqueName: \"kubernetes.io/projected/70694229-8fa5-4e3f-af7f-f875faba310b-kube-api-access-sk4hf\") pod \"maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.653804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.652561 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:35.936904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.936869 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97bd4c3a-aaf9-4b7f-ab96-7017555a32e6" path="/var/lib/kubelet/pods/97bd4c3a-aaf9-4b7f-ab96-7017555a32e6/volumes" Oct 31 03:35:35.970922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:35.970888 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx"] Oct 31 03:35:35.972823 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:35.972788 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70694229_8fa5_4e3f_af7f_f875faba310b.slice/crio-4f8aad11d39dcfc9cf4fd4e88ea7df753a0771905af6b70704676f1897e2ef3e WatchSource:0}: Error finding container 4f8aad11d39dcfc9cf4fd4e88ea7df753a0771905af6b70704676f1897e2ef3e: Status 404 returned error can't find the container with id 4f8aad11d39dcfc9cf4fd4e88ea7df753a0771905af6b70704676f1897e2ef3e Oct 31 03:35:36.896313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:36.896124 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" event={"ID":"70694229-8fa5-4e3f-af7f-f875faba310b","Type":"ContainerStarted","Data":"a75a0a11a273bb31127ff8de1aef0416a664c8018ffefb998969f92a80aecbcb"} Oct 31 03:35:36.896313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:36.896193 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" event={"ID":"70694229-8fa5-4e3f-af7f-f875faba310b","Type":"ContainerStarted","Data":"4f8aad11d39dcfc9cf4fd4e88ea7df753a0771905af6b70704676f1897e2ef3e"} Oct 31 03:35:36.896313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:36.896266 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:36.917726 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:36.917662 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" podStartSLOduration=1.4072372450000001 podStartE2EDuration="1.917640468s" totalImagesPullingTime="510.403223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:36.914051005 +0000 UTC m=+1161.108574127" watchObservedRunningTime="2026-10-31 03:35:36.917640468 +0000 UTC m=+1161.112163592" Oct 31 03:35:37.618449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:37.618410 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:35:37.820854 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:37.820813 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" podUID="da2b4e6d-7217-43b7-8154-9a417b58a292" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:35:37.821722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:37.821695 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" podUID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:35:39.359256 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:39.359218 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:35:40.948094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.948059 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:35:40.961452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.961418 5520 generic.go:356] "Generic (PLEG): container finished" podID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" containerID="9af914b16a13e41f65d4cc70119a3e41e81cae92ad7b65faee0cb5baa0ee2d3e" exitCode=0 Oct 31 03:35:40.961959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.961555 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" event={"ID":"993d5e91-a531-47a9-b3b2-48e66b40f1ac","Type":"ContainerDied","Data":"9af914b16a13e41f65d4cc70119a3e41e81cae92ad7b65faee0cb5baa0ee2d3e"} Oct 31 03:35:40.965408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.965380 5520 generic.go:356] "Generic (PLEG): container finished" podID="da2b4e6d-7217-43b7-8154-9a417b58a292" containerID="5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c" exitCode=0 Oct 31 03:35:40.965628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.965605 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" event={"ID":"da2b4e6d-7217-43b7-8154-9a417b58a292","Type":"ContainerDied","Data":"5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c"} Oct 31 03:35:40.965834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.965796 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" event={"ID":"da2b4e6d-7217-43b7-8154-9a417b58a292","Type":"ContainerDied","Data":"8faff16e7b97661f92141ca39bda89b325e34e85e30ae7a96d4514a9094a6107"} Oct 31 03:35:40.965939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.965858 5520 scope.go:122] "RemoveContainer" containerID="5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c" Oct 31 03:35:40.966098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.966083 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4" Oct 31 03:35:40.996450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.996330 5520 scope.go:122] "RemoveContainer" containerID="5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c" Oct 31 03:35:40.997043 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:40.996949 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c\": container with ID starting with 5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c not found: ID does not exist" containerID="5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c" Oct 31 03:35:40.997043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:40.996991 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c"} err="failed to get container status \"5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c\": rpc error: code = NotFound desc = could not find container \"5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c\": container with ID starting with 5c465685ca1caab6b6ce35b136330ea7bc4bb37ff2fcc8cfbd926eb52640963c not found: ID does not exist" Oct 31 03:35:41.045972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.045939 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2b4e6d-7217-43b7-8154-9a417b58a292-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2b4e6d-7217-43b7-8154-9a417b58a292-plugins-config-volume\") pod \"da2b4e6d-7217-43b7-8154-9a417b58a292\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " Oct 31 03:35:41.046494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.045998 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:35:41.046629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.046498 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2b4e6d-7217-43b7-8154-9a417b58a292-plugins-config-volume" pod "da2b4e6d-7217-43b7-8154-9a417b58a292" (UID: "da2b4e6d-7217-43b7-8154-9a417b58a292"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:41.046741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.046723 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2b4e6d-7217-43b7-8154-9a417b58a292-kube-api-access-b2czm\" (UniqueName: \"kubernetes.io/projected/da2b4e6d-7217-43b7-8154-9a417b58a292-kube-api-access-b2czm\") pod \"da2b4e6d-7217-43b7-8154-9a417b58a292\" (UID: \"da2b4e6d-7217-43b7-8154-9a417b58a292\") " Oct 31 03:35:41.047938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.047916 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2b4e6d-7217-43b7-8154-9a417b58a292-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:41.058495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.051334 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2b4e6d-7217-43b7-8154-9a417b58a292-kube-api-access-b2czm" pod "da2b4e6d-7217-43b7-8154-9a417b58a292" (UID: "da2b4e6d-7217-43b7-8154-9a417b58a292"). InnerVolumeSpecName "kube-api-access-b2czm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:41.082227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.082137 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx"] Oct 31 03:35:41.086230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.083221 5520 cpu_manager.go:414] "containerMap: removing container" podUID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" containerName="payload-pre-processing" Oct 31 03:35:41.086230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.083244 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" containerName="payload-pre-processing" Oct 31 03:35:41.086230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.083286 5520 cpu_manager.go:414] "containerMap: removing container" podUID="da2b4e6d-7217-43b7-8154-9a417b58a292" containerName="payload-processing" Oct 31 03:35:41.086230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.083295 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2b4e6d-7217-43b7-8154-9a417b58a292" containerName="payload-processing" Oct 31 03:35:41.086230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.083529 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" containerName="payload-pre-processing" Oct 31 03:35:41.086230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.083545 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2b4e6d-7217-43b7-8154-9a417b58a292" containerName="payload-processing" Oct 31 03:35:41.108229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.097833 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:41.108229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.105687 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-50d2d14c-dockercfg-9dls8\"" Oct 31 03:35:41.121490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.118455 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx"] Oct 31 03:35:41.146228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.145010 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w"] Oct 31 03:35:41.151228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.149937 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993d5e91-a531-47a9-b3b2-48e66b40f1ac-kube-api-access-xf44t\" (UniqueName: \"kubernetes.io/projected/993d5e91-a531-47a9-b3b2-48e66b40f1ac-kube-api-access-xf44t\") pod \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " Oct 31 03:35:41.151228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.150125 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993d5e91-a531-47a9-b3b2-48e66b40f1ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993d5e91-a531-47a9-b3b2-48e66b40f1ac-plugins-config-volume\") pod \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\" (UID: \"993d5e91-a531-47a9-b3b2-48e66b40f1ac\") " Oct 31 03:35:41.151228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.150849 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2czm\" (UniqueName: \"kubernetes.io/projected/da2b4e6d-7217-43b7-8154-9a417b58a292-kube-api-access-b2czm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:41.155245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.151973 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993d5e91-a531-47a9-b3b2-48e66b40f1ac-plugins-config-volume" pod "993d5e91-a531-47a9-b3b2-48e66b40f1ac" (UID: "993d5e91-a531-47a9-b3b2-48e66b40f1ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:35:41.155245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.154932 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993d5e91-a531-47a9-b3b2-48e66b40f1ac-kube-api-access-xf44t" pod "993d5e91-a531-47a9-b3b2-48e66b40f1ac" (UID: "993d5e91-a531-47a9-b3b2-48e66b40f1ac"). InnerVolumeSpecName "kube-api-access-xf44t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:35:41.158862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.158838 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w"] Oct 31 03:35:41.158975 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.158949 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.357230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.356307 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ee44b-36e7-4c0c-9a50-ec354e233fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:41.357230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.356507 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjhs\" (UniqueName: \"kubernetes.io/projected/277ee44b-36e7-4c0c-9a50-ec354e233fa0-kube-api-access-jxjhs\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:41.357230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.356627 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993d5e91-a531-47a9-b3b2-48e66b40f1ac-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:41.357230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.356641 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf44t\" (UniqueName: \"kubernetes.io/projected/993d5e91-a531-47a9-b3b2-48e66b40f1ac-kube-api-access-xf44t\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:35:41.471239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.458510 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ee44b-36e7-4c0c-9a50-ec354e233fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:41.471239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.458634 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jrp\" (UniqueName: \"kubernetes.io/projected/b6099aba-d9a2-4fc4-990a-ced0070bf14f-kube-api-access-w9jrp\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.471239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.458705 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6099aba-d9a2-4fc4-990a-ced0070bf14f-plugins-config-volume\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.471239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.458756 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjhs\" (UniqueName: \"kubernetes.io/projected/277ee44b-36e7-4c0c-9a50-ec354e233fa0-kube-api-access-jxjhs\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:41.471239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.460157 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ee44b-36e7-4c0c-9a50-ec354e233fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:41.492739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.492709 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjhs\" (UniqueName: \"kubernetes.io/projected/277ee44b-36e7-4c0c-9a50-ec354e233fa0-kube-api-access-jxjhs\") pod \"payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:41.578150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.560522 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jrp\" (UniqueName: \"kubernetes.io/projected/b6099aba-d9a2-4fc4-990a-ced0070bf14f-kube-api-access-w9jrp\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.578150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.560628 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6099aba-d9a2-4fc4-990a-ced0070bf14f-plugins-config-volume\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.578150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.571040 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6099aba-d9a2-4fc4-990a-ced0070bf14f-plugins-config-volume\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.611226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.610681 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jrp\" (UniqueName: \"kubernetes.io/projected/b6099aba-d9a2-4fc4-990a-ced0070bf14f-kube-api-access-w9jrp\") pod \"payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.686039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.686002 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4"] Oct 31 03:35:41.693196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.693159 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-pfkd4"] Oct 31 03:35:41.703784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.703754 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:41.714900 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:35:41.714877 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2b4e6d_7217_43b7_8154_9a417b58a292.slice/crio-8faff16e7b97661f92141ca39bda89b325e34e85e30ae7a96d4514a9094a6107\": RecentStats: unable to find data in memory cache]" Oct 31 03:35:41.734701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:41.734674 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:42.019797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.019718 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2b4e6d-7217-43b7-8154-9a417b58a292" path="/var/lib/kubelet/pods/da2b4e6d-7217-43b7-8154-9a417b58a292/volumes" Oct 31 03:35:42.029351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.024796 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" event={"ID":"993d5e91-a531-47a9-b3b2-48e66b40f1ac","Type":"ContainerDied","Data":"738012399e9fd881b9d8fa56511974d118507b953ac630fb63ae6f76228aaae0"} Oct 31 03:35:42.029351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.024866 5520 scope.go:122] "RemoveContainer" containerID="9af914b16a13e41f65d4cc70119a3e41e81cae92ad7b65faee0cb5baa0ee2d3e" Oct 31 03:35:42.029351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.025046 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj" Oct 31 03:35:42.029351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.028935 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w"] Oct 31 03:35:42.054876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.054843 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx"] Oct 31 03:35:42.055145 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:42.055124 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277ee44b_36e7_4c0c_9a50_ec354e233fa0.slice/crio-ee4f49fa7f444ee43a100f3ced8cdab95c773b47e1c4e513027a69a1f117ad01 WatchSource:0}: Error finding container ee4f49fa7f444ee43a100f3ced8cdab95c773b47e1c4e513027a69a1f117ad01: Status 404 returned error can't find the container with id ee4f49fa7f444ee43a100f3ced8cdab95c773b47e1c4e513027a69a1f117ad01 Oct 31 03:35:42.070262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.066569 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj"] Oct 31 03:35:42.082017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.081985 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-tmwsj"] Oct 31 03:35:42.936308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:42.934999 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:35:43.035842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.035806 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" event={"ID":"b6099aba-d9a2-4fc4-990a-ced0070bf14f","Type":"ContainerStarted","Data":"c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c"} Oct 31 03:35:43.036350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.035862 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" event={"ID":"b6099aba-d9a2-4fc4-990a-ced0070bf14f","Type":"ContainerStarted","Data":"4754792c3f5b3b6ae54fcf2bd9743c45e8d3d007c91f83a759414f758a5f5cb6"} Oct 31 03:35:43.036350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.035891 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:35:43.037972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.037944 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" event={"ID":"277ee44b-36e7-4c0c-9a50-ec354e233fa0","Type":"ContainerStarted","Data":"e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f"} Oct 31 03:35:43.038268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.038246 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" event={"ID":"277ee44b-36e7-4c0c-9a50-ec354e233fa0","Type":"ContainerStarted","Data":"ee4f49fa7f444ee43a100f3ced8cdab95c773b47e1c4e513027a69a1f117ad01"} Oct 31 03:35:43.038423 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.038401 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:43.054262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.054186 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" podStartSLOduration=2.054168587 podStartE2EDuration="2.054168587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:43.051493206 +0000 UTC m=+1167.246016351" watchObservedRunningTime="2026-10-31 03:35:43.054168587 +0000 UTC m=+1167.248691709" Oct 31 03:35:43.074690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.073935 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" podStartSLOduration=2.073917359 podStartE2EDuration="2.073917359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:43.069836478 +0000 UTC m=+1167.264359628" watchObservedRunningTime="2026-10-31 03:35:43.073917359 +0000 UTC m=+1167.268440485" Oct 31 03:35:43.919819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:43.919781 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993d5e91-a531-47a9-b3b2-48e66b40f1ac" path="/var/lib/kubelet/pods/993d5e91-a531-47a9-b3b2-48e66b40f1ac/volumes" Oct 31 03:35:44.809021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.808981 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2"] Oct 31 03:35:44.818682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.818656 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.833987 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.833958 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-50d2d14c-serving-cert\"" Oct 31 03:35:44.842368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.842336 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2"] Oct 31 03:35:44.873734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.873692 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdfp\" (UniqueName: \"kubernetes.io/projected/2f7190b2-fa95-40f2-9726-235132c22faf-kube-api-access-9sdfp\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.873905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.873851 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-maas-api-tls\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.873986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.873969 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-metrics-certs\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.874047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.874033 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7190b2-fa95-40f2-9726-235132c22faf-openshift-service-ca\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.976126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.976087 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdfp\" (UniqueName: \"kubernetes.io/projected/2f7190b2-fa95-40f2-9726-235132c22faf-kube-api-access-9sdfp\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.976429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.976319 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-maas-api-tls\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.976517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.976436 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-metrics-certs\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.976580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.976536 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7190b2-fa95-40f2-9726-235132c22faf-openshift-service-ca\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.977173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.977082 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7190b2-fa95-40f2-9726-235132c22faf-openshift-service-ca\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.978966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.978943 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-metrics-certs\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.979131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.979108 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-maas-api-tls\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:44.985077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:44.985052 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdfp\" (UniqueName: \"kubernetes.io/projected/2f7190b2-fa95-40f2-9726-235132c22faf-kube-api-access-9sdfp\") pod \"maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:45.159989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:45.159902 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:46.093697 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:35:46.093319 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7190b2_fa95_40f2_9726_235132c22faf.slice/crio-e850b54d54cf34eaf0a7994b6262a21c2ef08b0bdfb4b9bd506318997e723819 WatchSource:0}: Error finding container e850b54d54cf34eaf0a7994b6262a21c2ef08b0bdfb4b9bd506318997e723819: Status 404 returned error can't find the container with id e850b54d54cf34eaf0a7994b6262a21c2ef08b0bdfb4b9bd506318997e723819 Oct 31 03:35:46.097471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:46.097433 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2"] Oct 31 03:35:46.127958 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:46.127917 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" event={"ID":"2f7190b2-fa95-40f2-9726-235132c22faf","Type":"ContainerStarted","Data":"e850b54d54cf34eaf0a7994b6262a21c2ef08b0bdfb4b9bd506318997e723819"} Oct 31 03:35:47.137118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:47.137082 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" event={"ID":"2f7190b2-fa95-40f2-9726-235132c22faf","Type":"ContainerStarted","Data":"33c7eb02f0afda7c5525dabef4eeddd2699e4458be901d14e7c6e7a3185bdead"} Oct 31 03:35:47.137666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:47.137144 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:47.195626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:47.195564 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" podStartSLOduration=2.765378861 podStartE2EDuration="3.19554353s" totalImagesPullingTime="430.164669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:47.190054463 +0000 UTC m=+1171.384577619" watchObservedRunningTime="2026-10-31 03:35:47.19554353 +0000 UTC m=+1171.390066653" Oct 31 03:35:48.033238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:48.032286 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw"] Oct 31 03:35:48.052952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:48.050981 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6096acc7-vkqzw"] Oct 31 03:35:49.922532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:49.922478 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1950d237-6669-414e-a9d6-e16c4c1a75e3" path="/var/lib/kubelet/pods/1950d237-6669-414e-a9d6-e16c4c1a75e3/volumes" Oct 31 03:35:53.164526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:53.164482 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:35:54.068399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:54.066294 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:35:54.082570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:35:54.082536 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:36:03.138260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.138133 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7261-004b66"] Oct 31 03:36:03.142788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.142762 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:03.146517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.146493 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xd6kv\"" Oct 31 03:36:03.151004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.150980 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7261-004b66"] Oct 31 03:36:03.238969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.238925 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28g2\" (UniqueName: \"kubernetes.io/projected/3a4b362e-6fc6-4a11-85f9-64c739444074-kube-api-access-s28g2\") pod \"test-curl-7261-004b66\" (UID: \"3a4b362e-6fc6-4a11-85f9-64c739444074\") " pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:03.340892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.340859 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28g2\" (UniqueName: \"kubernetes.io/projected/3a4b362e-6fc6-4a11-85f9-64c739444074-kube-api-access-s28g2\") pod \"test-curl-7261-004b66\" (UID: \"3a4b362e-6fc6-4a11-85f9-64c739444074\") " pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:03.349282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.349247 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28g2\" (UniqueName: \"kubernetes.io/projected/3a4b362e-6fc6-4a11-85f9-64c739444074-kube-api-access-s28g2\") pod \"test-curl-7261-004b66\" (UID: \"3a4b362e-6fc6-4a11-85f9-64c739444074\") " pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:03.465031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.465001 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:03.828361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:03.826929 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7261-004b66"] Oct 31 03:36:04.668692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:04.668587 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7261-004b66" event={"ID":"3a4b362e-6fc6-4a11-85f9-64c739444074","Type":"ContainerStarted","Data":"462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b"} Oct 31 03:36:04.670694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:04.670669 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7261-004b66" event={"ID":"3a4b362e-6fc6-4a11-85f9-64c739444074","Type":"ContainerStarted","Data":"9c0463560ae7d5313d4c17f5f19e9062a0a20b749f24ad5130d31f54e6c472de"} Oct 31 03:36:04.724849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:04.724779 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7261-004b66" podStartSLOduration=1.7247581969999999 podStartE2EDuration="1.724758197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:04.707809341 +0000 UTC m=+1188.902332483" watchObservedRunningTime="2026-10-31 03:36:04.724758197 +0000 UTC m=+1188.919281319" Oct 31 03:36:06.006102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:06.006067 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7261-004b66"] Oct 31 03:36:06.521069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:06.521038 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7261-004b66"] Oct 31 03:36:06.795412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:06.795317 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:36:06.796653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:06.796399 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7261-004b66" podUID="3a4b362e-6fc6-4a11-85f9-64c739444074" containerName="test-curl-7261-004b66" containerID="cri-o://462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b" gracePeriod=2 Oct 31 03:36:06.803449 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:06.803411 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7261-004b66\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="3a4b362e-6fc6-4a11-85f9-64c739444074" pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:06.823659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:06.823575 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:06.825480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:06.825271 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:06.958989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:06.958949 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:09.330807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.330671 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:09.360479 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:09.360445 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:36:09.449242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.442430 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4b362e-6fc6-4a11-85f9-64c739444074-kube-api-access-s28g2\" (UniqueName: \"kubernetes.io/projected/3a4b362e-6fc6-4a11-85f9-64c739444074-kube-api-access-s28g2\") pod \"3a4b362e-6fc6-4a11-85f9-64c739444074\" (UID: \"3a4b362e-6fc6-4a11-85f9-64c739444074\") " Oct 31 03:36:09.457305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.454802 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4b362e-6fc6-4a11-85f9-64c739444074-kube-api-access-s28g2" pod "3a4b362e-6fc6-4a11-85f9-64c739444074" (UID: "3a4b362e-6fc6-4a11-85f9-64c739444074"). InnerVolumeSpecName "kube-api-access-s28g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:09.555346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.549839 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s28g2\" (UniqueName: \"kubernetes.io/projected/3a4b362e-6fc6-4a11-85f9-64c739444074-kube-api-access-s28g2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:09.858840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.858707 5520 generic.go:356] "Generic (PLEG): container finished" podID="3a4b362e-6fc6-4a11-85f9-64c739444074" containerID="462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b" exitCode=137 Oct 31 03:36:09.858840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.858819 5520 scope.go:122] "RemoveContainer" containerID="462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b" Oct 31 03:36:09.859081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.858823 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:09.887229 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:09.886492 5520 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7261-004b66\" is forbidden: User \"system:node:ip-10-0-141-153.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-153.us-east-2.compute.internal' and this object" podUID="3a4b362e-6fc6-4a11-85f9-64c739444074" pod="opendatahub/test-curl-7261-004b66" Oct 31 03:36:09.901505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.901468 5520 scope.go:122] "RemoveContainer" containerID="462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b" Oct 31 03:36:09.904548 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:09.904521 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b\": container with ID starting with 462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b not found: ID does not exist" containerID="462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b" Oct 31 03:36:09.904665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.904561 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b"} err="failed to get container status \"462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b\": rpc error: code = NotFound desc = could not find container \"462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b\": container with ID starting with 462ad12e8b8084c1311aded4df3429857bdecccd2e52ee299e0907325593e11b not found: ID does not exist" Oct 31 03:36:09.920784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:09.920745 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4b362e-6fc6-4a11-85f9-64c739444074" path="/var/lib/kubelet/pods/3a4b362e-6fc6-4a11-85f9-64c739444074/volumes" Oct 31 03:36:11.032247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:11.032213 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr"] Oct 31 03:36:11.036083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:11.036058 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31167051-hlwfr"] Oct 31 03:36:11.920030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:11.919995 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0bb301-2675-41e8-ae70-d69ad5709b0d" path="/var/lib/kubelet/pods/3a0bb301-2675-41e8-ae70-d69ad5709b0d/volumes" Oct 31 03:36:12.918337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:12.918308 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:36:12.918804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:12.918783 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:16.351999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:16.351960 5520 scope.go:122] "RemoveContainer" containerID="e16a69508e84f56a4aaf9527bb336b380e1ca275ee9152f17e8130987bb57c74" Oct 31 03:36:16.404244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:16.403674 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w"] Oct 31 03:36:16.426747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:16.426715 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx"] Oct 31 03:36:17.001227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:16.995179 5520 generic.go:356] "Generic (PLEG): container finished" podID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerID="84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9" exitCode=0 Oct 31 03:36:17.001227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:16.995367 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" event={"ID":"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7","Type":"ContainerDied","Data":"84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9"} Oct 31 03:36:17.001227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:16.995941 5520 scope.go:122] "RemoveContainer" containerID="84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9" Oct 31 03:36:17.022224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:17.021615 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:36:17.022224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:17.021723 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:36:17.022224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:17.021776 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:36:18.010709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:18.010673 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" event={"ID":"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7","Type":"ContainerStarted","Data":"4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7"} Oct 31 03:36:18.100226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:18.099364 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:36:18.100226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:18.099654 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:36:18.100226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:18.099706 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:36:19.102230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:19.100549 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:36:19.102230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:19.100617 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:36:20.103705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:20.103656 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:36:20.104133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:20.103733 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:36:21.099596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.099556 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:36:21.099925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.099897 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:36:21.405279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.404774 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" podUID="b6099aba-d9a2-4fc4-990a-ced0070bf14f" containerName="payload-processing" containerID="cri-o://c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c" gracePeriod=25 Oct 31 03:36:21.430706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.429973 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" podUID="277ee44b-36e7-4c0c-9a50-ec354e233fa0" containerName="payload-pre-processing" containerID="cri-o://e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f" gracePeriod=25 Oct 31 03:36:21.822613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.822586 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:36:21.927021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.924894 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6099aba-d9a2-4fc4-990a-ced0070bf14f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6099aba-d9a2-4fc4-990a-ced0070bf14f-plugins-config-volume\") pod \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " Oct 31 03:36:21.927021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.925162 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6099aba-d9a2-4fc4-990a-ced0070bf14f-kube-api-access-w9jrp\" (UniqueName: \"kubernetes.io/projected/b6099aba-d9a2-4fc4-990a-ced0070bf14f-kube-api-access-w9jrp\") pod \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\" (UID: \"b6099aba-d9a2-4fc4-990a-ced0070bf14f\") " Oct 31 03:36:21.927021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.925952 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6099aba-d9a2-4fc4-990a-ced0070bf14f-plugins-config-volume" pod "b6099aba-d9a2-4fc4-990a-ced0070bf14f" (UID: "b6099aba-d9a2-4fc4-990a-ced0070bf14f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:21.928989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.928963 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6099aba-d9a2-4fc4-990a-ced0070bf14f-kube-api-access-w9jrp" pod "b6099aba-d9a2-4fc4-990a-ced0070bf14f" (UID: "b6099aba-d9a2-4fc4-990a-ced0070bf14f"). InnerVolumeSpecName "kube-api-access-w9jrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:21.958467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:21.958445 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:36:22.026873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.026782 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277ee44b-36e7-4c0c-9a50-ec354e233fa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ee44b-36e7-4c0c-9a50-ec354e233fa0-plugins-config-volume\") pod \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " Oct 31 03:36:22.027044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.027018 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277ee44b-36e7-4c0c-9a50-ec354e233fa0-kube-api-access-jxjhs\" (UniqueName: \"kubernetes.io/projected/277ee44b-36e7-4c0c-9a50-ec354e233fa0-kube-api-access-jxjhs\") pod \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\" (UID: \"277ee44b-36e7-4c0c-9a50-ec354e233fa0\") " Oct 31 03:36:22.027876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.027828 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6099aba-d9a2-4fc4-990a-ced0070bf14f-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:22.027876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.027865 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jrp\" (UniqueName: \"kubernetes.io/projected/b6099aba-d9a2-4fc4-990a-ced0070bf14f-kube-api-access-w9jrp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:22.028741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.028710 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277ee44b-36e7-4c0c-9a50-ec354e233fa0-plugins-config-volume" pod "277ee44b-36e7-4c0c-9a50-ec354e233fa0" (UID: "277ee44b-36e7-4c0c-9a50-ec354e233fa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:22.031635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.031163 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277ee44b-36e7-4c0c-9a50-ec354e233fa0-kube-api-access-jxjhs" pod "277ee44b-36e7-4c0c-9a50-ec354e233fa0" (UID: "277ee44b-36e7-4c0c-9a50-ec354e233fa0"). InnerVolumeSpecName "kube-api-access-jxjhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:22.128292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.100724 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:36:22.128292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.100783 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:36:22.153250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.152025 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjhs\" (UniqueName: \"kubernetes.io/projected/277ee44b-36e7-4c0c-9a50-ec354e233fa0-kube-api-access-jxjhs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:22.153250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.152092 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ee44b-36e7-4c0c-9a50-ec354e233fa0-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.204953 5520 generic.go:356] "Generic (PLEG): container finished" podID="b6099aba-d9a2-4fc4-990a-ced0070bf14f" containerID="c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c" exitCode=0 Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.205032 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" event={"ID":"b6099aba-d9a2-4fc4-990a-ced0070bf14f","Type":"ContainerDied","Data":"c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c"} Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.205088 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" event={"ID":"b6099aba-d9a2-4fc4-990a-ced0070bf14f","Type":"ContainerDied","Data":"4754792c3f5b3b6ae54fcf2bd9743c45e8d3d007c91f83a759414f758a5f5cb6"} Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.205116 5520 scope.go:122] "RemoveContainer" containerID="c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c" Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.205298 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w" Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.208001 5520 generic.go:356] "Generic (PLEG): container finished" podID="277ee44b-36e7-4c0c-9a50-ec354e233fa0" containerID="e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f" exitCode=0 Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.208183 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" event={"ID":"277ee44b-36e7-4c0c-9a50-ec354e233fa0","Type":"ContainerDied","Data":"e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f"} Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.208694 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" Oct 31 03:36:22.215004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.212327 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx" event={"ID":"277ee44b-36e7-4c0c-9a50-ec354e233fa0","Type":"ContainerDied","Data":"ee4f49fa7f444ee43a100f3ced8cdab95c773b47e1c4e513027a69a1f117ad01"} Oct 31 03:36:22.252060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.252005 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w"] Oct 31 03:36:22.253667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.252406 5520 scope.go:122] "RemoveContainer" containerID="c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c" Oct 31 03:36:22.253667 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:22.252797 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c\": container with ID starting with c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c not found: ID does not exist" containerID="c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c" Oct 31 03:36:22.253667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.252843 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c"} err="failed to get container status \"c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c\": rpc error: code = NotFound desc = could not find container \"c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c\": container with ID starting with c6fa06fb272da59029062dd184cd3e9eb18a6cc8b22221e1b8c775fb4ab0234c not found: ID does not exist" Oct 31 03:36:22.253667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.252866 5520 scope.go:122] "RemoveContainer" containerID="e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f" Oct 31 03:36:22.260940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.260809 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50d2d14c-7b5d9b94b5-bll8w"] Oct 31 03:36:22.266875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.266848 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx"] Oct 31 03:36:22.272380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.270033 5520 scope.go:122] "RemoveContainer" containerID="e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f" Oct 31 03:36:22.286660 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:22.274686 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f\": container with ID starting with e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f not found: ID does not exist" containerID="e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f" Oct 31 03:36:22.286660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.274728 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f"} err="failed to get container status \"e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f\": rpc error: code = NotFound desc = could not find container \"e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f\": container with ID starting with e64a4ce7c28bf0c0b109bc23b007249dcd3c8ff4565962b8b445ad68afe0be1f not found: ID does not exist" Oct 31 03:36:22.286660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:22.282971 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50d2d14c-f6b76dbc-k4nrx"] Oct 31 03:36:23.099331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:23.099273 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:36:23.099841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:23.099339 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:36:23.427259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:23.427153 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2"] Oct 31 03:36:23.427692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:23.427660 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" podUID="2f7190b2-fa95-40f2-9726-235132c22faf" containerName="maas-api" containerID="cri-o://33c7eb02f0afda7c5525dabef4eeddd2699e4458be901d14e7c6e7a3185bdead" gracePeriod=30 Oct 31 03:36:24.019271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.002947 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277ee44b-36e7-4c0c-9a50-ec354e233fa0" path="/var/lib/kubelet/pods/277ee44b-36e7-4c0c-9a50-ec354e233fa0/volumes" Oct 31 03:36:24.051041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.051006 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6099aba-d9a2-4fc4-990a-ced0070bf14f" path="/var/lib/kubelet/pods/b6099aba-d9a2-4fc4-990a-ced0070bf14f/volumes" Oct 31 03:36:24.107632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.107596 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 31 03:36:24.108260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.108171 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 31 03:36:24.314361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.314326 5520 generic.go:356] "Generic (PLEG): container finished" podID="2f7190b2-fa95-40f2-9726-235132c22faf" containerID="33c7eb02f0afda7c5525dabef4eeddd2699e4458be901d14e7c6e7a3185bdead" exitCode=0 Oct 31 03:36:24.314828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.314809 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" event={"ID":"2f7190b2-fa95-40f2-9726-235132c22faf","Type":"ContainerDied","Data":"33c7eb02f0afda7c5525dabef4eeddd2699e4458be901d14e7c6e7a3185bdead"} Oct 31 03:36:24.582276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.582240 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:36:24.683404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.683359 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-maas-api-tls\") pod \"2f7190b2-fa95-40f2-9726-235132c22faf\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " Oct 31 03:36:24.683611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.683416 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7190b2-fa95-40f2-9726-235132c22faf-kube-api-access-9sdfp\" (UniqueName: \"kubernetes.io/projected/2f7190b2-fa95-40f2-9726-235132c22faf-kube-api-access-9sdfp\") pod \"2f7190b2-fa95-40f2-9726-235132c22faf\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " Oct 31 03:36:24.683611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.683539 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-metrics-certs\") pod \"2f7190b2-fa95-40f2-9726-235132c22faf\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " Oct 31 03:36:24.683611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.683593 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7190b2-fa95-40f2-9726-235132c22faf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7190b2-fa95-40f2-9726-235132c22faf-openshift-service-ca\") pod \"2f7190b2-fa95-40f2-9726-235132c22faf\" (UID: \"2f7190b2-fa95-40f2-9726-235132c22faf\") " Oct 31 03:36:24.684882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.684844 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7190b2-fa95-40f2-9726-235132c22faf-openshift-service-ca" pod "2f7190b2-fa95-40f2-9726-235132c22faf" (UID: "2f7190b2-fa95-40f2-9726-235132c22faf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:36:24.687613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.687546 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-metrics-certs" pod "2f7190b2-fa95-40f2-9726-235132c22faf" (UID: "2f7190b2-fa95-40f2-9726-235132c22faf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:24.687613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.687575 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7190b2-fa95-40f2-9726-235132c22faf-kube-api-access-9sdfp" pod "2f7190b2-fa95-40f2-9726-235132c22faf" (UID: "2f7190b2-fa95-40f2-9726-235132c22faf"). InnerVolumeSpecName "kube-api-access-9sdfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:24.689050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.689020 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-maas-api-tls" pod "2f7190b2-fa95-40f2-9726-235132c22faf" (UID: "2f7190b2-fa95-40f2-9726-235132c22faf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:24.823800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.804829 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:24.823800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.804865 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7190b2-fa95-40f2-9726-235132c22faf-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:24.823800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.804882 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7190b2-fa95-40f2-9726-235132c22faf-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:24.823800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:24.804896 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdfp\" (UniqueName: \"kubernetes.io/projected/2f7190b2-fa95-40f2-9726-235132c22faf-kube-api-access-9sdfp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:25.104628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.104538 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:36:25.104972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.104953 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:36:25.105785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.105765 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:36:25.323572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.323536 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" Oct 31 03:36:25.323572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.323563 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2" event={"ID":"2f7190b2-fa95-40f2-9726-235132c22faf","Type":"ContainerDied","Data":"e850b54d54cf34eaf0a7994b6262a21c2ef08b0bdfb4b9bd506318997e723819"} Oct 31 03:36:25.324125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.323630 5520 scope.go:122] "RemoveContainer" containerID="33c7eb02f0afda7c5525dabef4eeddd2699e4458be901d14e7c6e7a3185bdead" Oct 31 03:36:25.348411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.348383 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2"] Oct 31 03:36:25.352243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:25.352215 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50d2d14c-f4db9bdc9-g8cc2"] Oct 31 03:36:26.153331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:26.116187 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7190b2-fa95-40f2-9726-235132c22faf" path="/var/lib/kubelet/pods/2f7190b2-fa95-40f2-9726-235132c22faf/volumes" Oct 31 03:36:29.216948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:29.216916 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8"] Oct 31 03:36:31.114739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.114706 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6"] Oct 31 03:36:31.115781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115757 5520 cpu_manager.go:414] "containerMap: removing container" podUID="2f7190b2-fa95-40f2-9726-235132c22faf" containerName="maas-api" Oct 31 03:36:31.115904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115780 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7190b2-fa95-40f2-9726-235132c22faf" containerName="maas-api" Oct 31 03:36:31.115904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115833 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3a4b362e-6fc6-4a11-85f9-64c739444074" containerName="test-curl-7261-004b66" Oct 31 03:36:31.115904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115842 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4b362e-6fc6-4a11-85f9-64c739444074" containerName="test-curl-7261-004b66" Oct 31 03:36:31.115904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115873 5520 cpu_manager.go:414] "containerMap: removing container" podUID="277ee44b-36e7-4c0c-9a50-ec354e233fa0" containerName="payload-pre-processing" Oct 31 03:36:31.115904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115880 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277ee44b-36e7-4c0c-9a50-ec354e233fa0" containerName="payload-pre-processing" Oct 31 03:36:31.116256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115915 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b6099aba-d9a2-4fc4-990a-ced0070bf14f" containerName="payload-processing" Oct 31 03:36:31.116256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.115922 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6099aba-d9a2-4fc4-990a-ced0070bf14f" containerName="payload-processing" Oct 31 03:36:31.116256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.116106 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4b362e-6fc6-4a11-85f9-64c739444074" containerName="test-curl-7261-004b66" Oct 31 03:36:31.116256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.116130 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="277ee44b-36e7-4c0c-9a50-ec354e233fa0" containerName="payload-pre-processing" Oct 31 03:36:31.116256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.116152 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6099aba-d9a2-4fc4-990a-ced0070bf14f" containerName="payload-processing" Oct 31 03:36:31.116256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.116169 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7190b2-fa95-40f2-9726-235132c22faf" containerName="maas-api" Oct 31 03:36:31.121615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.121594 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.126499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.126175 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fc226c-dockercfg-4vnzf\"" Oct 31 03:36:31.130315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.130287 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6"] Oct 31 03:36:31.164226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.164180 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2"] Oct 31 03:36:31.169386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.169357 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:31.182030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.182001 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cd2b30-4669-4778-bf32-43e9cc4fc1de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.182286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.182105 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbcn9\" (UniqueName: \"kubernetes.io/projected/94cd2b30-4669-4778-bf32-43e9cc4fc1de-kube-api-access-sbcn9\") pod \"payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.191080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.190120 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2"] Oct 31 03:36:31.228769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.225636 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fc226c\"" Oct 31 03:36:31.420103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.420002 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrjq\" (UniqueName: \"kubernetes.io/projected/7862625a-9016-41f2-8fc8-082e9ec34a48-kube-api-access-ttrjq\") pod \"payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:31.420645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.420620 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7862625a-9016-41f2-8fc8-082e9ec34a48-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:31.420859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.420828 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cd2b30-4669-4778-bf32-43e9cc4fc1de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.522694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.422099 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cd2b30-4669-4778-bf32-43e9cc4fc1de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.539543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.531532 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbcn9\" (UniqueName: \"kubernetes.io/projected/94cd2b30-4669-4778-bf32-43e9cc4fc1de-kube-api-access-sbcn9\") pod \"payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.595394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.595356 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbcn9\" (UniqueName: \"kubernetes.io/projected/94cd2b30-4669-4778-bf32-43e9cc4fc1de-kube-api-access-sbcn9\") pod \"payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.681416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.681372 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrjq\" (UniqueName: \"kubernetes.io/projected/7862625a-9016-41f2-8fc8-082e9ec34a48-kube-api-access-ttrjq\") pod \"payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:31.681626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.681554 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7862625a-9016-41f2-8fc8-082e9ec34a48-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:31.682309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.682283 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7862625a-9016-41f2-8fc8-082e9ec34a48-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:31.689569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.689540 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrjq\" (UniqueName: \"kubernetes.io/projected/7862625a-9016-41f2-8fc8-082e9ec34a48-kube-api-access-ttrjq\") pod \"payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:31.752363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.752322 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:31.880943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:31.880784 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:32.054498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.053175 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6"] Oct 31 03:36:32.318847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.318811 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2"] Oct 31 03:36:32.319729 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:36:32.319702 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7862625a_9016_41f2_8fc8_082e9ec34a48.slice/crio-18fd7e72802753581b11e035ecf17934e527d0b168089148d1f2473850d96113 WatchSource:0}: Error finding container 18fd7e72802753581b11e035ecf17934e527d0b168089148d1f2473850d96113: Status 404 returned error can't find the container with id 18fd7e72802753581b11e035ecf17934e527d0b168089148d1f2473850d96113 Oct 31 03:36:32.589765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.589657 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" event={"ID":"94cd2b30-4669-4778-bf32-43e9cc4fc1de","Type":"ContainerStarted","Data":"5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711"} Oct 31 03:36:32.589765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.589732 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:32.589765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.589760 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" event={"ID":"94cd2b30-4669-4778-bf32-43e9cc4fc1de","Type":"ContainerStarted","Data":"d03d50151ea67d0da6a31d8fab50a07aa408635193c7fc34e80cf74f8d5d37ea"} Oct 31 03:36:32.591642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.591615 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" event={"ID":"7862625a-9016-41f2-8fc8-082e9ec34a48","Type":"ContainerStarted","Data":"a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e"} Oct 31 03:36:32.591774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.591666 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" event={"ID":"7862625a-9016-41f2-8fc8-082e9ec34a48","Type":"ContainerStarted","Data":"18fd7e72802753581b11e035ecf17934e527d0b168089148d1f2473850d96113"} Oct 31 03:36:32.591774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.591695 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:32.610852 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.610793 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" podStartSLOduration=1.610775297 podStartE2EDuration="1.610775297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:32.610050214 +0000 UTC m=+1216.804573339" watchObservedRunningTime="2026-10-31 03:36:32.610775297 +0000 UTC m=+1216.805298419" Oct 31 03:36:32.628742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:32.628695 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" podStartSLOduration=1.628680948 podStartE2EDuration="1.628680948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:32.625698272 +0000 UTC m=+1216.820221411" watchObservedRunningTime="2026-10-31 03:36:32.628680948 +0000 UTC m=+1216.823204069" Oct 31 03:36:39.360179 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:39.360142 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:36:40.686116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.686079 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656"] Oct 31 03:36:40.717853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.717824 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656"] Oct 31 03:36:40.718035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.717934 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.724070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.724040 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fc226c-serving-cert\"" Oct 31 03:36:40.813095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.813057 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-metrics-certs\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.813436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.813415 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtf4\" (UniqueName: \"kubernetes.io/projected/311a29cd-0635-4cd6-8ed7-ef31f7f80134-kube-api-access-7rtf4\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.813518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.813481 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311a29cd-0635-4cd6-8ed7-ef31f7f80134-openshift-service-ca\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.813685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.813662 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-maas-api-tls\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.917471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.917433 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-maas-api-tls\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.917781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.917762 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-metrics-certs\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.918153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.918130 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtf4\" (UniqueName: \"kubernetes.io/projected/311a29cd-0635-4cd6-8ed7-ef31f7f80134-kube-api-access-7rtf4\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.918339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.918322 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311a29cd-0635-4cd6-8ed7-ef31f7f80134-openshift-service-ca\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.919386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.919363 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311a29cd-0635-4cd6-8ed7-ef31f7f80134-openshift-service-ca\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.960138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.960058 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-maas-api-tls\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:40.963930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.963903 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-metrics-certs\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:41.000262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:40.998120 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtf4\" (UniqueName: \"kubernetes.io/projected/311a29cd-0635-4cd6-8ed7-ef31f7f80134-kube-api-access-7rtf4\") pod \"maas-api-e2e-api-a-fc226c-5b4849848f-fx656\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:41.035528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:41.035491 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:41.444302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:41.444263 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656"] Oct 31 03:36:41.733102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:41.733007 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" event={"ID":"311a29cd-0635-4cd6-8ed7-ef31f7f80134","Type":"ContainerStarted","Data":"6ce47b66116193cee1ea0e283dff53311ba9064e228ef91fb0e08fc3959b7e6a"} Oct 31 03:36:42.742601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:42.742559 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" event={"ID":"311a29cd-0635-4cd6-8ed7-ef31f7f80134","Type":"ContainerStarted","Data":"2d5c9f2ad0c7778f651b35a3c020bb71bc9374fd63dc35c7162d983703f05a7b"} Oct 31 03:36:42.743181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:42.742698 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:36:42.761691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:42.761639 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" podStartSLOduration=2.19919463 podStartE2EDuration="2.761625453s" totalImagesPullingTime="562.430823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:42.758917659 +0000 UTC m=+1226.953440797" watchObservedRunningTime="2026-10-31 03:36:42.761625453 +0000 UTC m=+1226.956148574" Oct 31 03:36:43.621284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:43.618765 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:36:43.621284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:43.620316 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:36:44.381758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.381702 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerName="main" containerID="cri-o://c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8" gracePeriod=45 Oct 31 03:36:44.467290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.466402 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": EOF" Oct 31 03:36:44.742227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.740921 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:36:44.816767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.816731 5520 generic.go:356] "Generic (PLEG): container finished" podID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerID="c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8" exitCode=0 Oct 31 03:36:44.819174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.817066 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" Oct 31 03:36:44.819850 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.818837 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" event={"ID":"8a50bdd8-10c1-47d2-b43f-91274a06c4f9","Type":"ContainerDied","Data":"c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8"} Oct 31 03:36:44.828597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.827301 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8" event={"ID":"8a50bdd8-10c1-47d2-b43f-91274a06c4f9","Type":"ContainerDied","Data":"00cc76f39cf844d72cb13fef3fc273c5eb486c7e72330ddc81f3729c49afd133"} Oct 31 03:36:44.828597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.827348 5520 scope.go:122] "RemoveContainer" containerID="c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8" Oct 31 03:36:44.845777 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.845757 5520 scope.go:122] "RemoveContainer" containerID="c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8" Oct 31 03:36:44.846099 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:36:44.846073 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8\": container with ID starting with c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8 not found: ID does not exist" containerID="c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8" Oct 31 03:36:44.846235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.846113 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8"} err="failed to get container status \"c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8\": rpc error: code = NotFound desc = could not find container \"c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8\": container with ID starting with c899feb70a6268af8f65f44bb07fc58842b36b0fb2a56be0213738f52ef5c7e8 not found: ID does not exist" Oct 31 03:36:44.861919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.861889 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-model-cache\") pod \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " Oct 31 03:36:44.862069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.861971 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tmp-dir\") pod \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " Oct 31 03:36:44.862069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.862008 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-home\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-home\") pod \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " Oct 31 03:36:44.862270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.862142 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-kube-api-access-w54gs\" (UniqueName: \"kubernetes.io/projected/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-kube-api-access-w54gs\") pod \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " Oct 31 03:36:44.862270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.862265 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tls-certs\") pod \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " Oct 31 03:36:44.862386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.862340 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-dshm\") pod \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\" (UID: \"8a50bdd8-10c1-47d2-b43f-91274a06c4f9\") " Oct 31 03:36:44.863272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.863248 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-home" pod "8a50bdd8-10c1-47d2-b43f-91274a06c4f9" (UID: "8a50bdd8-10c1-47d2-b43f-91274a06c4f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:44.863408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.863394 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-model-cache" pod "8a50bdd8-10c1-47d2-b43f-91274a06c4f9" (UID: "8a50bdd8-10c1-47d2-b43f-91274a06c4f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:44.863547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.863531 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tmp-dir" pod "8a50bdd8-10c1-47d2-b43f-91274a06c4f9" (UID: "8a50bdd8-10c1-47d2-b43f-91274a06c4f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:44.866760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.866729 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tls-certs" pod "8a50bdd8-10c1-47d2-b43f-91274a06c4f9" (UID: "8a50bdd8-10c1-47d2-b43f-91274a06c4f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:36:44.866951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.866926 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-kube-api-access-w54gs" pod "8a50bdd8-10c1-47d2-b43f-91274a06c4f9" (UID: "8a50bdd8-10c1-47d2-b43f-91274a06c4f9"). InnerVolumeSpecName "kube-api-access-w54gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:36:44.867067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.867047 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-dshm" pod "8a50bdd8-10c1-47d2-b43f-91274a06c4f9" (UID: "8a50bdd8-10c1-47d2-b43f-91274a06c4f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:36:44.964981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.964949 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w54gs\" (UniqueName: \"kubernetes.io/projected/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-kube-api-access-w54gs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.964981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.964978 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.964981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.964988 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.965260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.965000 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.965260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.965012 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:44.965260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:44.965021 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a50bdd8-10c1-47d2-b43f-91274a06c4f9-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:36:45.246349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:45.241261 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8"] Oct 31 03:36:45.267233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:45.260389 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-50d2d14c/e2e-lifecycle-model-50d2d14c-kserve-f94dd85b9-98tc8"] Oct 31 03:36:45.933308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:45.933267 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" path="/var/lib/kubelet/pods/8a50bdd8-10c1-47d2-b43f-91274a06c4f9/volumes" Oct 31 03:36:48.759405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:36:48.759376 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:37:01.634641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.634606 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx"] Oct 31 03:37:01.635394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.635378 5520 cpu_manager.go:414] "containerMap: removing container" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerName="main" Oct 31 03:37:01.635447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.635397 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerName="main" Oct 31 03:37:01.635568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.635558 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a50bdd8-10c1-47d2-b43f-91274a06c4f9" containerName="main" Oct 31 03:37:01.640806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.640785 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:01.645746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.645721 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b87769\"" Oct 31 03:37:01.645746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.645736 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b87769-dockercfg-2hfq2\"" Oct 31 03:37:01.652273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.652196 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx"] Oct 31 03:37:01.659342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.659318 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x"] Oct 31 03:37:01.696971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.696921 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x"] Oct 31 03:37:01.697133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.697034 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:01.736636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.736602 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6z56\" (UniqueName: \"kubernetes.io/projected/00d47495-d197-4a35-a556-fdd27b25658a-kube-api-access-j6z56\") pod \"payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:01.736950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.736917 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf84c\" (UniqueName: \"kubernetes.io/projected/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-kube-api-access-kf84c\") pod \"payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:01.737142 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.737070 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:01.737142 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.737102 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d47495-d197-4a35-a556-fdd27b25658a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:01.857229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.841478 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6z56\" (UniqueName: \"kubernetes.io/projected/00d47495-d197-4a35-a556-fdd27b25658a-kube-api-access-j6z56\") pod \"payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:01.857229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.841553 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf84c\" (UniqueName: \"kubernetes.io/projected/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-kube-api-access-kf84c\") pod \"payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:01.857229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.841623 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:01.857229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.841648 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d47495-d197-4a35-a556-fdd27b25658a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:01.857229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.842791 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d47495-d197-4a35-a556-fdd27b25658a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:01.857229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.853051 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:01.890278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.886981 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf84c\" (UniqueName: \"kubernetes.io/projected/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-kube-api-access-kf84c\") pod \"payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:01.915226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:01.891136 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6z56\" (UniqueName: \"kubernetes.io/projected/00d47495-d197-4a35-a556-fdd27b25658a-kube-api-access-j6z56\") pod \"payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:02.036231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:02.026368 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:02.259611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:02.259577 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:02.494966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:02.494433 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x"] Oct 31 03:37:02.507258 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:02.507099 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9a7093_aaee_4d4e_b8d1_1d8f957e1c38.slice/crio-1cc55af2a56db8c2cf353e318f5286b4abb7df17460183a23cdaf1b218599f3f WatchSource:0}: Error finding container 1cc55af2a56db8c2cf353e318f5286b4abb7df17460183a23cdaf1b218599f3f: Status 404 returned error can't find the container with id 1cc55af2a56db8c2cf353e318f5286b4abb7df17460183a23cdaf1b218599f3f Oct 31 03:37:02.573542 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:02.573508 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx"] Oct 31 03:37:03.192012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.191943 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" event={"ID":"00d47495-d197-4a35-a556-fdd27b25658a","Type":"ContainerStarted","Data":"c69ff1f7dc09f7a1a1ea40d2cceb5039f5a5200e12d168608c239583106e4a9a"} Oct 31 03:37:03.192512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.192054 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" event={"ID":"00d47495-d197-4a35-a556-fdd27b25658a","Type":"ContainerStarted","Data":"fe6b2a9507bbb521992ba8008ac20ed6b46a9556c6e0c44cc22448fe08586b3a"} Oct 31 03:37:03.192512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.192081 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:03.193034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.193002 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" event={"ID":"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38","Type":"ContainerStarted","Data":"e64ee0379bfd6c2850dea66d02ebf4421c197ada27f739f19bbc743a511a63fb"} Oct 31 03:37:03.193100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.193066 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:03.193100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.193090 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" event={"ID":"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38","Type":"ContainerStarted","Data":"1cc55af2a56db8c2cf353e318f5286b4abb7df17460183a23cdaf1b218599f3f"} Oct 31 03:37:03.211055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.211010 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" podStartSLOduration=2.210995742 podStartE2EDuration="2.210995742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:03.209616989 +0000 UTC m=+1247.404140136" watchObservedRunningTime="2026-10-31 03:37:03.210995742 +0000 UTC m=+1247.405518862" Oct 31 03:37:03.230624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:03.230577 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" podStartSLOduration=2.230562769 podStartE2EDuration="2.230562769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:03.228438077 +0000 UTC m=+1247.422961244" watchObservedRunningTime="2026-10-31 03:37:03.230562769 +0000 UTC m=+1247.425085904" Oct 31 03:37:09.362011 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:09.361973 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:37:11.391183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.391142 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw"] Oct 31 03:37:11.404898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.404869 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw"] Oct 31 03:37:11.405071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.404986 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:11.423122 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.423089 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-afb8be-dockercfg-wthjg\"" Oct 31 03:37:11.467172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.467144 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-afb8be\"" Oct 31 03:37:11.526231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.505118 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqf7\" (UniqueName: \"kubernetes.io/projected/50b13be8-fc9d-4cac-9193-658b2bb30eae-kube-api-access-glqf7\") pod \"payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:11.527938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.527903 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b13be8-fc9d-4cac-9193-658b2bb30eae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:11.630804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.630767 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b13be8-fc9d-4cac-9193-658b2bb30eae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:11.631541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.631503 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqf7\" (UniqueName: \"kubernetes.io/projected/50b13be8-fc9d-4cac-9193-658b2bb30eae-kube-api-access-glqf7\") pod \"payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:11.633313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.633290 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b13be8-fc9d-4cac-9193-658b2bb30eae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:11.687252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.683791 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqf7\" (UniqueName: \"kubernetes.io/projected/50b13be8-fc9d-4cac-9193-658b2bb30eae-kube-api-access-glqf7\") pod \"payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:11.736486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:11.736447 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:12.114342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:12.112807 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw"] Oct 31 03:37:12.543765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:12.543725 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" event={"ID":"50b13be8-fc9d-4cac-9193-658b2bb30eae","Type":"ContainerStarted","Data":"a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4"} Oct 31 03:37:12.544376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:12.543794 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" event={"ID":"50b13be8-fc9d-4cac-9193-658b2bb30eae","Type":"ContainerStarted","Data":"04b2375475aea7423483d0adcf6825a23b9576a8c2ce683ae8a46a3f460abae7"} Oct 31 03:37:12.544376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:12.543987 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:12.571379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:12.571322 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" podStartSLOduration=1.5713021139999999 podStartE2EDuration="1.571302114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:12.562933248 +0000 UTC m=+1256.757456395" watchObservedRunningTime="2026-10-31 03:37:12.571302114 +0000 UTC m=+1256.765825236" Oct 31 03:37:14.216227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:14.214885 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:37:14.216227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:14.215435 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:37:15.093572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:15.093542 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5"] Oct 31 03:37:15.094140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:15.094112 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" containerID="cri-o://401b4295ce180f76c7a4b1786fb3b292d7f2666b064e7a4da3f89da849e74f81" gracePeriod=30 Oct 31 03:37:16.415223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:16.415156 5520 scope.go:122] "RemoveContainer" containerID="781ba11ef52787d5b456acb6eed2490e624ecccb904f04a3f7ee40a901f29457" Oct 31 03:37:16.438738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:16.438710 5520 scope.go:122] "RemoveContainer" containerID="bca9ccc2d11c2315bb17bb65b9dfa4e099ae5f51c0621cbe17d07e0351ed96fc" Oct 31 03:37:19.685016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:19.684982 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47"] Oct 31 03:37:19.700283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:19.700247 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2"] Oct 31 03:37:20.659575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.659538 5520 generic.go:356] "Generic (PLEG): container finished" podID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerID="401b4295ce180f76c7a4b1786fb3b292d7f2666b064e7a4da3f89da849e74f81" exitCode=0 Oct 31 03:37:20.659773 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.659615 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" event={"ID":"25c3c6f6-df1a-4885-96ed-5a840a5f41c6","Type":"ContainerDied","Data":"401b4295ce180f76c7a4b1786fb3b292d7f2666b064e7a4da3f89da849e74f81"} Oct 31 03:37:20.659773 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.659727 5520 scope.go:122] "RemoveContainer" containerID="8c4a7e96b51370e34d1bcb220d9c66165402fc62d665343cec970e598188ffa9" Oct 31 03:37:20.686772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.685353 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll"] Oct 31 03:37:20.686772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.685680 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" containerID="cri-o://55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d" gracePeriod=30 Oct 31 03:37:20.765819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.765789 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:37:20.910117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.910030 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-kube-api-access-srhh5\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-kube-api-access-srhh5\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.910340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.910194 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-podinfo\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.910340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.910262 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-data\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.910340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.910295 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-token\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-token\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.910522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.910424 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-envoy\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.910522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.910459 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-certs\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.910522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.910489 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-socket\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.911331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.911284 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-certs" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.911552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.911498 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-credential-socket\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.911636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.911551 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-socket" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.911800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.911777 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-credential-socket" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.911892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.911796 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-data" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.911892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.911853 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istiod-ca-cert\") pod \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\" (UID: \"25c3c6f6-df1a-4885-96ed-5a840a5f41c6\") " Oct 31 03:37:20.912187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.912163 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istiod-ca-cert" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:20.912860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.912833 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.912959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.912868 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.912959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.912883 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.912959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.912898 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.912959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.912912 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.922127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.922066 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-podinfo" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:20.922127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.922090 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-envoy" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.922323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.922160 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-token" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:20.922455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:20.922437 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-kube-api-access-srhh5" pod "25c3c6f6-df1a-4885-96ed-5a840a5f41c6" (UID: "25c3c6f6-df1a-4885-96ed-5a840a5f41c6"). InnerVolumeSpecName "kube-api-access-srhh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:21.023005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.018189 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhh5\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-kube-api-access-srhh5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:21.023005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.018248 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:21.023005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.018263 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:21.023005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.018278 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c3c6f6-df1a-4885-96ed-5a840a5f41c6-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:21.669166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.669140 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" Oct 31 03:37:21.669166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.669146 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5" event={"ID":"25c3c6f6-df1a-4885-96ed-5a840a5f41c6","Type":"ContainerDied","Data":"ae5ba28027bf6809f8bf59799a6bf1ee3484c8f235602be11148d5269d204940"} Oct 31 03:37:21.669479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.669222 5520 scope.go:122] "RemoveContainer" containerID="401b4295ce180f76c7a4b1786fb3b292d7f2666b064e7a4da3f89da849e74f81" Oct 31 03:37:21.699826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.699793 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5"] Oct 31 03:37:21.703686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.703659 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-46vx5"] Oct 31 03:37:21.919884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:21.919800 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" path="/var/lib/kubelet/pods/25c3c6f6-df1a-4885-96ed-5a840a5f41c6/volumes" Oct 31 03:37:23.554426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:23.554398 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:37:24.685568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:24.685525 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" podUID="5337a83c-9532-4771-b918-65770e4fa1cc" containerName="payload-processing" containerID="cri-o://cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a" gracePeriod=25 Oct 31 03:37:24.701007 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:24.700966 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" podUID="ad527969-608c-4bdb-afe8-a4fb138623b3" containerName="payload-pre-processing" containerID="cri-o://93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714" gracePeriod=25 Oct 31 03:37:25.019603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.019564 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:37:25.062817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.062786 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5337a83c-9532-4771-b918-65770e4fa1cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337a83c-9532-4771-b918-65770e4fa1cc-plugins-config-volume\") pod \"5337a83c-9532-4771-b918-65770e4fa1cc\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " Oct 31 03:37:25.062999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.062873 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5337a83c-9532-4771-b918-65770e4fa1cc-kube-api-access-6c9t2\" (UniqueName: \"kubernetes.io/projected/5337a83c-9532-4771-b918-65770e4fa1cc-kube-api-access-6c9t2\") pod \"5337a83c-9532-4771-b918-65770e4fa1cc\" (UID: \"5337a83c-9532-4771-b918-65770e4fa1cc\") " Oct 31 03:37:25.064928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.064879 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5337a83c-9532-4771-b918-65770e4fa1cc-plugins-config-volume" pod "5337a83c-9532-4771-b918-65770e4fa1cc" (UID: "5337a83c-9532-4771-b918-65770e4fa1cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:25.078493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.078463 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:37:25.081498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.081455 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5337a83c-9532-4771-b918-65770e4fa1cc-kube-api-access-6c9t2" pod "5337a83c-9532-4771-b918-65770e4fa1cc" (UID: "5337a83c-9532-4771-b918-65770e4fa1cc"). InnerVolumeSpecName "kube-api-access-6c9t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:25.165601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.165551 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad527969-608c-4bdb-afe8-a4fb138623b3-kube-api-access-zdsht\" (UniqueName: \"kubernetes.io/projected/ad527969-608c-4bdb-afe8-a4fb138623b3-kube-api-access-zdsht\") pod \"ad527969-608c-4bdb-afe8-a4fb138623b3\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " Oct 31 03:37:25.165823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.165735 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad527969-608c-4bdb-afe8-a4fb138623b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad527969-608c-4bdb-afe8-a4fb138623b3-plugins-config-volume\") pod \"ad527969-608c-4bdb-afe8-a4fb138623b3\" (UID: \"ad527969-608c-4bdb-afe8-a4fb138623b3\") " Oct 31 03:37:25.166109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.166084 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad527969-608c-4bdb-afe8-a4fb138623b3-plugins-config-volume" pod "ad527969-608c-4bdb-afe8-a4fb138623b3" (UID: "ad527969-608c-4bdb-afe8-a4fb138623b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:25.166386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.166368 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5337a83c-9532-4771-b918-65770e4fa1cc-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:25.166483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.166387 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9t2\" (UniqueName: \"kubernetes.io/projected/5337a83c-9532-4771-b918-65770e4fa1cc-kube-api-access-6c9t2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:25.166483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.166398 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad527969-608c-4bdb-afe8-a4fb138623b3-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:25.178184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.168304 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad527969-608c-4bdb-afe8-a4fb138623b3-kube-api-access-zdsht" pod "ad527969-608c-4bdb-afe8-a4fb138623b3" (UID: "ad527969-608c-4bdb-afe8-a4fb138623b3"). InnerVolumeSpecName "kube-api-access-zdsht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:25.273336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.268702 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdsht\" (UniqueName: \"kubernetes.io/projected/ad527969-608c-4bdb-afe8-a4fb138623b3-kube-api-access-zdsht\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:25.806707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.806452 5520 generic.go:356] "Generic (PLEG): container finished" podID="ad527969-608c-4bdb-afe8-a4fb138623b3" containerID="93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714" exitCode=0 Oct 31 03:37:25.806707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.806557 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" Oct 31 03:37:25.806707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.806683 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" event={"ID":"ad527969-608c-4bdb-afe8-a4fb138623b3","Type":"ContainerDied","Data":"93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714"} Oct 31 03:37:25.807347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.806764 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2" event={"ID":"ad527969-608c-4bdb-afe8-a4fb138623b3","Type":"ContainerDied","Data":"e35965297336c0737f878f34308805432174bcc9cc8266ccb2d2d9e4e7f4a387"} Oct 31 03:37:25.807347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.806800 5520 scope.go:122] "RemoveContainer" containerID="93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714" Oct 31 03:37:25.809702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.809678 5520 generic.go:356] "Generic (PLEG): container finished" podID="5337a83c-9532-4771-b918-65770e4fa1cc" containerID="cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a" exitCode=0 Oct 31 03:37:25.809901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.809883 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" Oct 31 03:37:25.810126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.810099 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" event={"ID":"5337a83c-9532-4771-b918-65770e4fa1cc","Type":"ContainerDied","Data":"cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a"} Oct 31 03:37:25.810340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.810321 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47" event={"ID":"5337a83c-9532-4771-b918-65770e4fa1cc","Type":"ContainerDied","Data":"4479fb67223cabff5b572f77657a7dde0f1db644d6a1af86fb42ea4c86b52cd2"} Oct 31 03:37:25.833307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.833286 5520 scope.go:122] "RemoveContainer" containerID="93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714" Oct 31 03:37:25.833600 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:25.833584 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714\": container with ID starting with 93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714 not found: ID does not exist" containerID="93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714" Oct 31 03:37:25.833650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.833610 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714"} err="failed to get container status \"93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714\": rpc error: code = NotFound desc = could not find container \"93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714\": container with ID starting with 93397210347a1db7d662ef36e9fbbeee2da33a9c0b8eca908e7bebdfa59bc714 not found: ID does not exist" Oct 31 03:37:25.833650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.833626 5520 scope.go:122] "RemoveContainer" containerID="cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a" Oct 31 03:37:25.842715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.842686 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2"] Oct 31 03:37:25.845115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.845093 5520 scope.go:122] "RemoveContainer" containerID="cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a" Oct 31 03:37:25.845812 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:25.845777 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a\": container with ID starting with cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a not found: ID does not exist" containerID="cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a" Oct 31 03:37:25.845932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.845818 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a"} err="failed to get container status \"cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a\": rpc error: code = NotFound desc = could not find container \"cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a\": container with ID starting with cfba9fadc5847d552f666cc602f9dad6ec8707f656f512387b6e875e5950703a not found: ID does not exist" Oct 31 03:37:25.846852 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.846830 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8afac225-5f55796b87-555s2"] Oct 31 03:37:25.858307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.858274 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47"] Oct 31 03:37:25.863392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:25.863366 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8afac225-cd74654c9-2fc47"] Oct 31 03:37:26.041230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.038117 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5337a83c-9532-4771-b918-65770e4fa1cc" path="/var/lib/kubelet/pods/5337a83c-9532-4771-b918-65770e4fa1cc/volumes" Oct 31 03:37:26.041230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.038963 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad527969-608c-4bdb-afe8-a4fb138623b3" path="/var/lib/kubelet/pods/ad527969-608c-4bdb-afe8-a4fb138623b3/volumes" Oct 31 03:37:26.407238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.407170 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:37:26.606358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606269 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-istio-token\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-istio-token\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606405 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-certs\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606543 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-data\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606608 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-kube-api-access-dfss9\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-kube-api-access-dfss9\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606703 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-credential-socket\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606752 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-socket\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606783 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42611746-9ded-4093-8e03-087bf1734ab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42611746-9ded-4093-8e03-087bf1734ab3-istiod-ca-cert\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606869 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-envoy\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.606943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.606906 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42611746-9ded-4093-8e03-087bf1734ab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42611746-9ded-4093-8e03-087bf1734ab3-istio-podinfo\") pod \"42611746-9ded-4093-8e03-087bf1734ab3\" (UID: \"42611746-9ded-4093-8e03-087bf1734ab3\") " Oct 31 03:37:26.607729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.607694 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-data" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.607941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.607890 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-socket" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.608117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.608096 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-credential-socket" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.608117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.608109 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42611746-9ded-4093-8e03-087bf1734ab3-istiod-ca-cert" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:26.616250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.610286 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-kube-api-access-dfss9" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "kube-api-access-dfss9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:26.616250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.610501 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-istio-token" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:26.616250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.610589 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42611746-9ded-4093-8e03-087bf1734ab3-istio-podinfo" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:26.616250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.611194 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-envoy" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.616250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.610079 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-certs" pod "42611746-9ded-4093-8e03-087bf1734ab3" (UID: "42611746-9ded-4093-8e03-087bf1734ab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.708862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708818 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfss9\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-kube-api-access-dfss9\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.708862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708854 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.708862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708868 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.709121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708881 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42611746-9ded-4093-8e03-087bf1734ab3-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.709121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708893 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.709121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708905 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42611746-9ded-4093-8e03-087bf1734ab3-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.709121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708916 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42611746-9ded-4093-8e03-087bf1734ab3-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.709121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708928 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.709121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.708940 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42611746-9ded-4093-8e03-087bf1734ab3-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.827452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.827417 5520 generic.go:356] "Generic (PLEG): container finished" podID="42611746-9ded-4093-8e03-087bf1734ab3" containerID="55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d" exitCode=0 Oct 31 03:37:26.827929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.827499 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" event={"ID":"42611746-9ded-4093-8e03-087bf1734ab3","Type":"ContainerDied","Data":"55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d"} Oct 31 03:37:26.827929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.827588 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" event={"ID":"42611746-9ded-4093-8e03-087bf1734ab3","Type":"ContainerDied","Data":"29d752c74d92ff0b6e321cb8fffa6d5a8d326c6b2bca173614ba072a53700aa0"} Oct 31 03:37:26.827929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.827622 5520 scope.go:122] "RemoveContainer" containerID="55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d" Oct 31 03:37:26.827929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.827794 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll" Oct 31 03:37:26.854410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.854385 5520 scope.go:122] "RemoveContainer" containerID="55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d" Oct 31 03:37:26.854885 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:26.854807 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d\": container with ID starting with 55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d not found: ID does not exist" containerID="55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d" Oct 31 03:37:26.854885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.854847 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d"} err="failed to get container status \"55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d\": rpc error: code = NotFound desc = could not find container \"55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d\": container with ID starting with 55de9b43cd63a594f0545f7d4871184701f52a5789e290298ea7bcd9fde5635d not found: ID does not exist" Oct 31 03:37:26.872308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.872231 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll"] Oct 31 03:37:26.878404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:26.878376 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-2wbll"] Oct 31 03:37:27.923111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:27.923066 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42611746-9ded-4093-8e03-087bf1734ab3" path="/var/lib/kubelet/pods/42611746-9ded-4093-8e03-087bf1734ab3/volumes" Oct 31 03:37:31.238270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.238226 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk"] Oct 31 03:37:31.238881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.238507 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" podUID="12490f36-1e53-40e8-b0bc-57c1995ba5df" containerName="maas-api" containerID="cri-o://2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f" gracePeriod=30 Oct 31 03:37:31.555988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.555961 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:37:31.775284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.774769 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-metrics-certs\") pod \"12490f36-1e53-40e8-b0bc-57c1995ba5df\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " Oct 31 03:37:31.775284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.774857 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12490f36-1e53-40e8-b0bc-57c1995ba5df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12490f36-1e53-40e8-b0bc-57c1995ba5df-openshift-service-ca\") pod \"12490f36-1e53-40e8-b0bc-57c1995ba5df\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " Oct 31 03:37:31.775284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.774996 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-maas-api-tls\") pod \"12490f36-1e53-40e8-b0bc-57c1995ba5df\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " Oct 31 03:37:31.775284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.775188 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12490f36-1e53-40e8-b0bc-57c1995ba5df-kube-api-access-xjvpk\" (UniqueName: \"kubernetes.io/projected/12490f36-1e53-40e8-b0bc-57c1995ba5df-kube-api-access-xjvpk\") pod \"12490f36-1e53-40e8-b0bc-57c1995ba5df\" (UID: \"12490f36-1e53-40e8-b0bc-57c1995ba5df\") " Oct 31 03:37:31.800232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.778410 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12490f36-1e53-40e8-b0bc-57c1995ba5df-openshift-service-ca" pod "12490f36-1e53-40e8-b0bc-57c1995ba5df" (UID: "12490f36-1e53-40e8-b0bc-57c1995ba5df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:31.819226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.813361 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-maas-api-tls" pod "12490f36-1e53-40e8-b0bc-57c1995ba5df" (UID: "12490f36-1e53-40e8-b0bc-57c1995ba5df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:31.819226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.813883 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-metrics-certs" pod "12490f36-1e53-40e8-b0bc-57c1995ba5df" (UID: "12490f36-1e53-40e8-b0bc-57c1995ba5df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:37:31.847538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.847502 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12490f36-1e53-40e8-b0bc-57c1995ba5df-kube-api-access-xjvpk" pod "12490f36-1e53-40e8-b0bc-57c1995ba5df" (UID: "12490f36-1e53-40e8-b0bc-57c1995ba5df"). InnerVolumeSpecName "kube-api-access-xjvpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:31.883703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.883601 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvpk\" (UniqueName: \"kubernetes.io/projected/12490f36-1e53-40e8-b0bc-57c1995ba5df-kube-api-access-xjvpk\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:31.883703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.883647 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:31.883703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.883660 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12490f36-1e53-40e8-b0bc-57c1995ba5df-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:31.883703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:31.883674 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12490f36-1e53-40e8-b0bc-57c1995ba5df-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:32.033544 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.033434 5520 generic.go:356] "Generic (PLEG): container finished" podID="12490f36-1e53-40e8-b0bc-57c1995ba5df" containerID="2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f" exitCode=0 Oct 31 03:37:32.047358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.047326 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" Oct 31 03:37:32.080412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.079745 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" event={"ID":"12490f36-1e53-40e8-b0bc-57c1995ba5df","Type":"ContainerDied","Data":"2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f"} Oct 31 03:37:32.080412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.079828 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk" event={"ID":"12490f36-1e53-40e8-b0bc-57c1995ba5df","Type":"ContainerDied","Data":"0227262aa67f09d44b9770f0175558b76e5931b72251461488d23533534e5330"} Oct 31 03:37:32.080412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.079872 5520 scope.go:122] "RemoveContainer" containerID="2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f" Oct 31 03:37:32.108403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.106789 5520 scope.go:122] "RemoveContainer" containerID="2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f" Oct 31 03:37:32.108403 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:32.107185 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f\": container with ID starting with 2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f not found: ID does not exist" containerID="2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f" Oct 31 03:37:32.108403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.107255 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f"} err="failed to get container status \"2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f\": rpc error: code = NotFound desc = could not find container \"2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f\": container with ID starting with 2fe064cea561ef9b177dbfc6c42baab950b7a11a893252f717798d5dcd55ba5f not found: ID does not exist" Oct 31 03:37:32.119703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.119674 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk"] Oct 31 03:37:32.123045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:32.123024 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8afac225-68647fb488-7qfdk"] Oct 31 03:37:33.950232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:33.943107 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12490f36-1e53-40e8-b0bc-57c1995ba5df" path="/var/lib/kubelet/pods/12490f36-1e53-40e8-b0bc-57c1995ba5df/volumes" Oct 31 03:37:35.884094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:35.884061 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt"] Oct 31 03:37:35.884573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:35.884342 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" containerID="cri-o://c456f6196a172ad408c109fe2bcdb0bfc7e913a721b008bb49c03888322873a2" gracePeriod=30 Oct 31 03:37:35.896897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:35.896866 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f"] Oct 31 03:37:35.897368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:35.897328 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" containerID="cri-o://61a8302d544c25e4fc2518d5eb8ff6761c681c1b60d5af54084394ed9d575a8f" gracePeriod=30 Oct 31 03:37:38.587851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:38.587821 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg"] Oct 31 03:37:38.588716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:38.588665 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" containerID="cri-o://4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7" gracePeriod=30 Oct 31 03:37:39.361281 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:39.361243 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:37:40.147248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:40.145880 5520 patch_prober.go:29] interesting pod/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:37:40.147248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:40.145948 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:37:41.274714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.274675 5520 generic.go:356] "Generic (PLEG): container finished" podID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerID="61a8302d544c25e4fc2518d5eb8ff6761c681c1b60d5af54084394ed9d575a8f" exitCode=0 Oct 31 03:37:41.278311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.275348 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" event={"ID":"2e460fac-4fc9-47ca-af24-19f08f093c46","Type":"ContainerDied","Data":"61a8302d544c25e4fc2518d5eb8ff6761c681c1b60d5af54084394ed9d575a8f"} Oct 31 03:37:41.281045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.281021 5520 generic.go:356] "Generic (PLEG): container finished" podID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerID="c456f6196a172ad408c109fe2bcdb0bfc7e913a721b008bb49c03888322873a2" exitCode=0 Oct 31 03:37:41.284839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.284564 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" event={"ID":"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7","Type":"ContainerDied","Data":"c456f6196a172ad408c109fe2bcdb0bfc7e913a721b008bb49c03888322873a2"} Oct 31 03:37:41.557857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.557824 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:37:41.565186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.565153 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-podinfo\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.565366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.565233 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-socket\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.565434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.565365 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-data\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.565492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.565443 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-token\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-token\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.565552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.565525 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-envoy\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.566357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.565662 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-credential-socket\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.566357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.565974 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e460fac-4fc9-47ca-af24-19f08f093c46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e460fac-4fc9-47ca-af24-19f08f093c46-istiod-ca-cert\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.566551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.566473 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-credential-socket" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.570032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.566710 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-kube-api-access-p9vh4\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-kube-api-access-p9vh4\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.570032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.566761 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-certs\") pod \"2e460fac-4fc9-47ca-af24-19f08f093c46\" (UID: \"2e460fac-4fc9-47ca-af24-19f08f093c46\") " Oct 31 03:37:41.570032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.566913 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e460fac-4fc9-47ca-af24-19f08f093c46-istiod-ca-cert" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:41.570032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.566958 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-data" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.570032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.568571 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-certs" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.570032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.569504 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-kube-api-access-p9vh4" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "kube-api-access-p9vh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:41.571135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.571092 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-socket" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.571255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.571230 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.571332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.571254 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e460fac-4fc9-47ca-af24-19f08f093c46-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.571332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.571270 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9vh4\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-kube-api-access-p9vh4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.571332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.571292 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.571332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.571303 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.573449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.573413 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-token" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:41.573564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.573520 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-envoy" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.573674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.573650 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-podinfo" pod "2e460fac-4fc9-47ca-af24-19f08f093c46" (UID: "2e460fac-4fc9-47ca-af24-19f08f093c46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:41.628427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.628388 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2"] Oct 31 03:37:41.659877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.649897 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6"] Oct 31 03:37:41.686251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.685978 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.686251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.686013 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.686251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.686030 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.686251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.686046 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e460fac-4fc9-47ca-af24-19f08f093c46-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.713470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.707954 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:37:41.788062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.787952 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-socket\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788074 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-credential-socket\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788114 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-kube-api-access-4d6gl\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-kube-api-access-4d6gl\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788159 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istiod-ca-cert\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788428 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-envoy\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788460 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-certs\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788511 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-podinfo\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788600 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-data\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.788689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.788642 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-token\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-token\") pod \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\" (UID: \"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7\") " Oct 31 03:37:41.790126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.790095 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istiod-ca-cert" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:41.790389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.790366 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-socket" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.790563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.790542 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-credential-socket" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.791344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.791312 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-data" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.791788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.791758 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-certs" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.814937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.814896 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-token" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:41.816384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.816356 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-envoy" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:41.816384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.816366 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-podinfo" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:41.818363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.818341 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-kube-api-access-4d6gl" pod "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" (UID: "14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7"). InnerVolumeSpecName "kube-api-access-4d6gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:41.891448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891413 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891441 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891451 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891461 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891470 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891480 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6gl\" (UniqueName: \"kubernetes.io/projected/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-kube-api-access-4d6gl\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891488 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891496 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:41.891686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:41.891504 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:42.367313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.365110 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" event={"ID":"2e460fac-4fc9-47ca-af24-19f08f093c46","Type":"ContainerDied","Data":"d359ec4371524439f0da8b6e0a6e80b594ef587db75e06d3d44877696f8c772e"} Oct 31 03:37:42.367313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.365174 5520 scope.go:122] "RemoveContainer" containerID="61a8302d544c25e4fc2518d5eb8ff6761c681c1b60d5af54084394ed9d575a8f" Oct 31 03:37:42.367313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.365362 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f" Oct 31 03:37:42.371623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.371582 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" event={"ID":"14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7","Type":"ContainerDied","Data":"54ddcaf10c4b08bb0eaf803c1115e33de306b46bda86b25b435c293f43bb5ef5"} Oct 31 03:37:42.371766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.371698 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt" Oct 31 03:37:42.400372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.400341 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt"] Oct 31 03:37:42.401943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.401921 5520 scope.go:122] "RemoveContainer" containerID="c456f6196a172ad408c109fe2bcdb0bfc7e913a721b008bb49c03888322873a2" Oct 31 03:37:42.414097 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.414063 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-l72jt"] Oct 31 03:37:42.426036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.425123 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f"] Oct 31 03:37:42.430064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:42.429888 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-6lp5f"] Oct 31 03:37:43.843092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:43.843054 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011347 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-socket\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011502 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-kube-api-access-vz7qv\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-kube-api-access-vz7qv\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011580 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-certs\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011620 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-token\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-token\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011651 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-envoy\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011707 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-credential-socket\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011777 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-data\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.011892 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istiod-ca-cert\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.012586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.012036 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-podinfo\") pod \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\" (UID: \"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7\") " Oct 31 03:37:44.013417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.013303 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istiod-ca-cert" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:44.013600 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.013558 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-credential-socket" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.013756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.013736 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-data" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.021173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.019360 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.021173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.019389 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.021173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.019405 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.021574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.021246 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-socket" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.021574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.021309 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-certs" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.035558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.035487 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-kube-api-access-vz7qv" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "kube-api-access-vz7qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:44.035815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.035781 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-envoy" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:44.068167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.060916 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-podinfo" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:44.068167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.063765 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-token" pod "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" (UID: "c1be6ef9-0699-4d3c-be31-9e93ace5bbe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:44.093369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.084080 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" path="/var/lib/kubelet/pods/14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7/volumes" Oct 31 03:37:44.093369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.084849 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" path="/var/lib/kubelet/pods/2e460fac-4fc9-47ca-af24-19f08f093c46/volumes" Oct 31 03:37:44.121647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.121610 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.121647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.121643 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.121875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.121659 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz7qv\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-kube-api-access-vz7qv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.121875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.121672 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.121875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.121680 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.121875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.121689 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:44.394770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.394665 5520 generic.go:356] "Generic (PLEG): container finished" podID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerID="4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7" exitCode=0 Oct 31 03:37:44.394770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.394741 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" Oct 31 03:37:44.394999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.394775 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" event={"ID":"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7","Type":"ContainerDied","Data":"4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7"} Oct 31 03:37:44.394999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.394846 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg" event={"ID":"c1be6ef9-0699-4d3c-be31-9e93ace5bbe7","Type":"ContainerDied","Data":"00b571c19f6db93ca0bcd0e7a98215daba083a0d5ace170d0c4c0b8e6bf4aad6"} Oct 31 03:37:44.394999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.394880 5520 scope.go:122] "RemoveContainer" containerID="4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7" Oct 31 03:37:44.406384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.406364 5520 scope.go:122] "RemoveContainer" containerID="84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9" Oct 31 03:37:44.419189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.419165 5520 scope.go:122] "RemoveContainer" containerID="4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7" Oct 31 03:37:44.419506 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:44.419485 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7\": container with ID starting with 4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7 not found: ID does not exist" containerID="4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7" Oct 31 03:37:44.419571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.419515 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7"} err="failed to get container status \"4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7\": rpc error: code = NotFound desc = could not find container \"4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7\": container with ID starting with 4320929f1763939f45641cfe4c2cb3c3ed17b0d44572f63077eccd398c4c8ca7 not found: ID does not exist" Oct 31 03:37:44.419571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.419533 5520 scope.go:122] "RemoveContainer" containerID="84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9" Oct 31 03:37:44.419783 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:44.419764 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9\": container with ID starting with 84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9 not found: ID does not exist" containerID="84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9" Oct 31 03:37:44.419832 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.419789 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9"} err="failed to get container status \"84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9\": rpc error: code = NotFound desc = could not find container \"84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9\": container with ID starting with 84c9bd9814c4fe199e190e79fa7fad2c92e4a4612ac724e17929bde89d9824d9 not found: ID does not exist" Oct 31 03:37:44.437902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.437871 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg"] Oct 31 03:37:44.446487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:44.446458 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8afac225-openshift-default-5bbfd94c69-m7jwg"] Oct 31 03:37:45.718683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.718651 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb"] Oct 31 03:37:45.719412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719397 5520 cpu_manager.go:414] "containerMap: removing container" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" Oct 31 03:37:45.719478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719415 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" Oct 31 03:37:45.719478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719432 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5337a83c-9532-4771-b918-65770e4fa1cc" containerName="payload-processing" Oct 31 03:37:45.719478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719437 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5337a83c-9532-4771-b918-65770e4fa1cc" containerName="payload-processing" Oct 31 03:37:45.719478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719453 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" Oct 31 03:37:45.719478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719459 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" Oct 31 03:37:45.719478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719471 5520 cpu_manager.go:414] "containerMap: removing container" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" Oct 31 03:37:45.719478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719476 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719502 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719511 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719524 5520 cpu_manager.go:414] "containerMap: removing container" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719529 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719542 5520 cpu_manager.go:414] "containerMap: removing container" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719546 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719553 5520 cpu_manager.go:414] "containerMap: removing container" podUID="12490f36-1e53-40e8-b0bc-57c1995ba5df" containerName="maas-api" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719558 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12490f36-1e53-40e8-b0bc-57c1995ba5df" containerName="maas-api" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719572 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ad527969-608c-4bdb-afe8-a4fb138623b3" containerName="payload-pre-processing" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719576 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad527969-608c-4bdb-afe8-a4fb138623b3" containerName="payload-pre-processing" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719678 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e460fac-4fc9-47ca-af24-19f08f093c46" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719689 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719697 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719703 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="42611746-9ded-4093-8e03-087bf1734ab3" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719708 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a8ffdd-6cda-4f06-aaa6-edaf38a7dab7" containerName="istio-proxy" Oct 31 03:37:45.719712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719715 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="12490f36-1e53-40e8-b0bc-57c1995ba5df" containerName="maas-api" Oct 31 03:37:45.720191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719722 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad527969-608c-4bdb-afe8-a4fb138623b3" containerName="payload-pre-processing" Oct 31 03:37:45.720191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719729 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" Oct 31 03:37:45.720191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.719735 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5337a83c-9532-4771-b918-65770e4fa1cc" containerName="payload-processing" Oct 31 03:37:45.724950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.724929 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:45.731512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.731483 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-adb7b7-dockercfg-ssd4q\"" Oct 31 03:37:45.731640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.731581 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-adb7b7\"" Oct 31 03:37:45.736096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.736052 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb"] Oct 31 03:37:45.772927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.772889 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc"] Oct 31 03:37:45.773874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.773851 5520 cpu_manager.go:414] "containerMap: removing container" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" Oct 31 03:37:45.773874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.773873 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c3c6f6-df1a-4885-96ed-5a840a5f41c6" containerName="istio-proxy" Oct 31 03:37:45.774048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.774034 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" containerName="istio-proxy" Oct 31 03:37:45.778079 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.778054 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:45.811248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.804677 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc"] Oct 31 03:37:45.861855 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.861824 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7j6\" (UniqueName: \"kubernetes.io/projected/c6143c03-8a03-4e2b-ac02-18ac593c103f-kube-api-access-qz7j6\") pod \"payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:45.862042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:45.861994 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6143c03-8a03-4e2b-ac02-18ac593c103f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:46.021282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.021059 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6143c03-8a03-4e2b-ac02-18ac593c103f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:46.021475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.021365 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891e5f8d-1360-406b-894d-8cb637655611-plugins-config-volume\") pod \"payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:46.026228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.021561 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7j6\" (UniqueName: \"kubernetes.io/projected/c6143c03-8a03-4e2b-ac02-18ac593c103f-kube-api-access-qz7j6\") pod \"payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:46.026228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.021651 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqfqs\" (UniqueName: \"kubernetes.io/projected/891e5f8d-1360-406b-894d-8cb637655611-kube-api-access-vqfqs\") pod \"payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:46.026228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.023945 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6143c03-8a03-4e2b-ac02-18ac593c103f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:46.079229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.073314 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1be6ef9-0699-4d3c-be31-9e93ace5bbe7" path="/var/lib/kubelet/pods/c1be6ef9-0699-4d3c-be31-9e93ace5bbe7/volumes" Oct 31 03:37:46.128226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.128019 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891e5f8d-1360-406b-894d-8cb637655611-plugins-config-volume\") pod \"payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:46.147366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.147325 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqfqs\" (UniqueName: \"kubernetes.io/projected/891e5f8d-1360-406b-894d-8cb637655611-kube-api-access-vqfqs\") pod \"payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:46.149033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.149005 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891e5f8d-1360-406b-894d-8cb637655611-plugins-config-volume\") pod \"payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:46.170550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.170191 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7j6\" (UniqueName: \"kubernetes.io/projected/c6143c03-8a03-4e2b-ac02-18ac593c103f-kube-api-access-qz7j6\") pod \"payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:46.201497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.199839 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqfqs\" (UniqueName: \"kubernetes.io/projected/891e5f8d-1360-406b-894d-8cb637655611-kube-api-access-vqfqs\") pod \"payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:46.343761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.343664 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:46.399383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.399339 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:46.629244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.629145 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" podUID="7862625a-9016-41f2-8fc8-082e9ec34a48" containerName="payload-processing" containerID="cri-o://a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e" gracePeriod=25 Oct 31 03:37:46.650292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.650258 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" podUID="94cd2b30-4669-4778-bf32-43e9cc4fc1de" containerName="payload-pre-processing" containerID="cri-o://5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711" gracePeriod=25 Oct 31 03:37:46.658313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.658279 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb"] Oct 31 03:37:46.682190 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:46.682159 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6143c03_8a03_4e2b_ac02_18ac593c103f.slice/crio-c62c57ea35b7c13f545e970970957529cd3d507f6ac7a3d52d160223c2763521 WatchSource:0}: Error finding container c62c57ea35b7c13f545e970970957529cd3d507f6ac7a3d52d160223c2763521: Status 404 returned error can't find the container with id c62c57ea35b7c13f545e970970957529cd3d507f6ac7a3d52d160223c2763521 Oct 31 03:37:46.732391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.732360 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc"] Oct 31 03:37:46.739498 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:46.739469 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891e5f8d_1360_406b_894d_8cb637655611.slice/crio-ffc21c493892f1d7a5f868c9bd806f44e8ebc87e9fcaf24239e5d451b411537f WatchSource:0}: Error finding container ffc21c493892f1d7a5f868c9bd806f44e8ebc87e9fcaf24239e5d451b411537f: Status 404 returned error can't find the container with id ffc21c493892f1d7a5f868c9bd806f44e8ebc87e9fcaf24239e5d451b411537f Oct 31 03:37:46.976334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:46.976057 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:37:47.017484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.017450 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj"] Oct 31 03:37:47.018455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.018433 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7862625a-9016-41f2-8fc8-082e9ec34a48" containerName="payload-processing" Oct 31 03:37:47.018455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.018457 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7862625a-9016-41f2-8fc8-082e9ec34a48" containerName="payload-processing" Oct 31 03:37:47.018742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.018722 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7862625a-9016-41f2-8fc8-082e9ec34a48" containerName="payload-processing" Oct 31 03:37:47.052232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.051003 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.067891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.067049 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-555bd6\"" Oct 31 03:37:47.068172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.068107 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj"] Oct 31 03:37:47.075042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.075019 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq"] Oct 31 03:37:47.100234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.094951 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq"] Oct 31 03:37:47.100234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.095115 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.180810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.162915 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-555bd6-dockercfg-djx8d\"" Oct 31 03:37:47.183863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.182281 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7862625a-9016-41f2-8fc8-082e9ec34a48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7862625a-9016-41f2-8fc8-082e9ec34a48-plugins-config-volume\") pod \"7862625a-9016-41f2-8fc8-082e9ec34a48\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " Oct 31 03:37:47.183863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.182622 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7862625a-9016-41f2-8fc8-082e9ec34a48-kube-api-access-ttrjq\" (UniqueName: \"kubernetes.io/projected/7862625a-9016-41f2-8fc8-082e9ec34a48-kube-api-access-ttrjq\") pod \"7862625a-9016-41f2-8fc8-082e9ec34a48\" (UID: \"7862625a-9016-41f2-8fc8-082e9ec34a48\") " Oct 31 03:37:47.183863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.183050 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.183863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.183370 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb96d9d-566b-4d00-9c59-857a2d53552f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.183863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.183493 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdfh\" (UniqueName: \"kubernetes.io/projected/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-kube-api-access-tbdfh\") pod \"payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.183863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.183530 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4x5q\" (UniqueName: \"kubernetes.io/projected/fdb96d9d-566b-4d00-9c59-857a2d53552f-kube-api-access-t4x5q\") pod \"payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.265225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.252773 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7862625a-9016-41f2-8fc8-082e9ec34a48-kube-api-access-ttrjq" pod "7862625a-9016-41f2-8fc8-082e9ec34a48" (UID: "7862625a-9016-41f2-8fc8-082e9ec34a48"). InnerVolumeSpecName "kube-api-access-ttrjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:47.294986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.294947 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7862625a-9016-41f2-8fc8-082e9ec34a48-plugins-config-volume" pod "7862625a-9016-41f2-8fc8-082e9ec34a48" (UID: "7862625a-9016-41f2-8fc8-082e9ec34a48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.317494 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.317873 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb96d9d-566b-4d00-9c59-857a2d53552f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.318006 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdfh\" (UniqueName: \"kubernetes.io/projected/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-kube-api-access-tbdfh\") pod \"payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.318103 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4x5q\" (UniqueName: \"kubernetes.io/projected/fdb96d9d-566b-4d00-9c59-857a2d53552f-kube-api-access-t4x5q\") pod \"payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.318252 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrjq\" (UniqueName: \"kubernetes.io/projected/7862625a-9016-41f2-8fc8-082e9ec34a48-kube-api-access-ttrjq\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.318273 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7862625a-9016-41f2-8fc8-082e9ec34a48-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.318680 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.319819 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb96d9d-566b-4d00-9c59-857a2d53552f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.332225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.320699 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.463298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.435032 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cd2b30-4669-4778-bf32-43e9cc4fc1de-kube-api-access-sbcn9\" (UniqueName: \"kubernetes.io/projected/94cd2b30-4669-4778-bf32-43e9cc4fc1de-kube-api-access-sbcn9\") pod \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " Oct 31 03:37:47.463298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.435164 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94cd2b30-4669-4778-bf32-43e9cc4fc1de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cd2b30-4669-4778-bf32-43e9cc4fc1de-plugins-config-volume\") pod \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\" (UID: \"94cd2b30-4669-4778-bf32-43e9cc4fc1de\") " Oct 31 03:37:47.466298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.465978 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4x5q\" (UniqueName: \"kubernetes.io/projected/fdb96d9d-566b-4d00-9c59-857a2d53552f-kube-api-access-t4x5q\") pod \"payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.487534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.485240 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cd2b30-4669-4778-bf32-43e9cc4fc1de-plugins-config-volume" pod "94cd2b30-4669-4778-bf32-43e9cc4fc1de" (UID: "94cd2b30-4669-4778-bf32-43e9cc4fc1de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:47.487534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.485379 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cd2b30-4669-4778-bf32-43e9cc4fc1de-kube-api-access-sbcn9" pod "94cd2b30-4669-4778-bf32-43e9cc4fc1de" (UID: "94cd2b30-4669-4778-bf32-43e9cc4fc1de"). InnerVolumeSpecName "kube-api-access-sbcn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:47.531655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.528851 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" event={"ID":"c6143c03-8a03-4e2b-ac02-18ac593c103f","Type":"ContainerStarted","Data":"07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d"} Oct 31 03:37:47.534427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.532418 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" event={"ID":"c6143c03-8a03-4e2b-ac02-18ac593c103f","Type":"ContainerStarted","Data":"c62c57ea35b7c13f545e970970957529cd3d507f6ac7a3d52d160223c2763521"} Oct 31 03:37:47.535619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.535590 5520 generic.go:356] "Generic (PLEG): container finished" podID="94cd2b30-4669-4778-bf32-43e9cc4fc1de" containerID="5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711" exitCode=0 Oct 31 03:37:47.535898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.535880 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" Oct 31 03:37:47.536291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.536260 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" event={"ID":"94cd2b30-4669-4778-bf32-43e9cc4fc1de","Type":"ContainerDied","Data":"5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711"} Oct 31 03:37:47.536415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.536318 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6" event={"ID":"94cd2b30-4669-4778-bf32-43e9cc4fc1de","Type":"ContainerDied","Data":"d03d50151ea67d0da6a31d8fab50a07aa408635193c7fc34e80cf74f8d5d37ea"} Oct 31 03:37:47.536415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.536353 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:47.536415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.536388 5520 scope.go:122] "RemoveContainer" containerID="5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711" Oct 31 03:37:47.571453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.571234 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:47.575607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.575577 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdfh\" (UniqueName: \"kubernetes.io/projected/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-kube-api-access-tbdfh\") pod \"payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.588462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.588365 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbcn9\" (UniqueName: \"kubernetes.io/projected/94cd2b30-4669-4778-bf32-43e9cc4fc1de-kube-api-access-sbcn9\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:47.588462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.588398 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cd2b30-4669-4778-bf32-43e9cc4fc1de-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:47.592863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.592829 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" event={"ID":"891e5f8d-1360-406b-894d-8cb637655611","Type":"ContainerStarted","Data":"beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d"} Oct 31 03:37:47.593087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.593069 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" event={"ID":"891e5f8d-1360-406b-894d-8cb637655611","Type":"ContainerStarted","Data":"ffc21c493892f1d7a5f868c9bd806f44e8ebc87e9fcaf24239e5d451b411537f"} Oct 31 03:37:47.594517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.594255 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:37:47.597964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.597940 5520 generic.go:356] "Generic (PLEG): container finished" podID="7862625a-9016-41f2-8fc8-082e9ec34a48" containerID="a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e" exitCode=0 Oct 31 03:37:47.598554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.598531 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" event={"ID":"7862625a-9016-41f2-8fc8-082e9ec34a48","Type":"ContainerDied","Data":"a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e"} Oct 31 03:37:47.598708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.598693 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" event={"ID":"7862625a-9016-41f2-8fc8-082e9ec34a48","Type":"ContainerDied","Data":"18fd7e72802753581b11e035ecf17934e527d0b168089148d1f2473850d96113"} Oct 31 03:37:47.598865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.598852 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2" Oct 31 03:37:47.598966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.598925 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" podStartSLOduration=2.598911425 podStartE2EDuration="2.598911425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:47.595555368 +0000 UTC m=+1291.790078501" watchObservedRunningTime="2026-10-31 03:37:47.598911425 +0000 UTC m=+1291.793434548" Oct 31 03:37:47.615480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.615429 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" podStartSLOduration=2.615411428 podStartE2EDuration="2.615411428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:47.614149995 +0000 UTC m=+1291.808673125" watchObservedRunningTime="2026-10-31 03:37:47.615411428 +0000 UTC m=+1291.809934551" Oct 31 03:37:47.629039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.629015 5520 scope.go:122] "RemoveContainer" containerID="5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711" Oct 31 03:37:47.629605 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:47.629575 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711\": container with ID starting with 5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711 not found: ID does not exist" containerID="5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711" Oct 31 03:37:47.629705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.629616 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711"} err="failed to get container status \"5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711\": rpc error: code = NotFound desc = could not find container \"5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711\": container with ID starting with 5bc1b9c747f11deac9b60a0a3e7523618711ef74e59f7fb362d4596ac3047711 not found: ID does not exist" Oct 31 03:37:47.629705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.629640 5520 scope.go:122] "RemoveContainer" containerID="a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e" Oct 31 03:37:47.643056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.643019 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6"] Oct 31 03:37:47.650618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.650592 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fc226c-757868b9ff-bcrz6"] Oct 31 03:37:47.655424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.653770 5520 scope.go:122] "RemoveContainer" containerID="a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e" Oct 31 03:37:47.655424 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:37:47.654105 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e\": container with ID starting with a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e not found: ID does not exist" containerID="a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e" Oct 31 03:37:47.655424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.654132 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e"} err="failed to get container status \"a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e\": rpc error: code = NotFound desc = could not find container \"a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e\": container with ID starting with a4bba0221d4642cfa1c3017bcaa4ef171a1825349ad706ce5702636c90b2c88e not found: ID does not exist" Oct 31 03:37:47.665633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.665605 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2"] Oct 31 03:37:47.670319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.670299 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fc226c-7bf5b89687-fjgp2"] Oct 31 03:37:47.833190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.833102 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:47.906442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.906414 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj"] Oct 31 03:37:47.911060 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:47.911025 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb96d9d_566b_4d00_9c59_857a2d53552f.slice/crio-15d828ba6a4304b2f5be130293adfbf8c89d6b5ca1e0d184af160ab5f9cc2a6c WatchSource:0}: Error finding container 15d828ba6a4304b2f5be130293adfbf8c89d6b5ca1e0d184af160ab5f9cc2a6c: Status 404 returned error can't find the container with id 15d828ba6a4304b2f5be130293adfbf8c89d6b5ca1e0d184af160ab5f9cc2a6c Oct 31 03:37:47.930001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.929974 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7862625a-9016-41f2-8fc8-082e9ec34a48" path="/var/lib/kubelet/pods/7862625a-9016-41f2-8fc8-082e9ec34a48/volumes" Oct 31 03:37:47.930848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:47.930828 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94cd2b30-4669-4778-bf32-43e9cc4fc1de" path="/var/lib/kubelet/pods/94cd2b30-4669-4778-bf32-43e9cc4fc1de/volumes" Oct 31 03:37:48.247532 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:48.247490 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bb7aaf_5529_4cd5_ace6_1f8e89b243c7.slice/crio-5fd7465a066b5d4c7306d6cd8316f09ab8957afa41a6a8207f160b95fadce318 WatchSource:0}: Error finding container 5fd7465a066b5d4c7306d6cd8316f09ab8957afa41a6a8207f160b95fadce318: Status 404 returned error can't find the container with id 5fd7465a066b5d4c7306d6cd8316f09ab8957afa41a6a8207f160b95fadce318 Oct 31 03:37:48.251918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.251890 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq"] Oct 31 03:37:48.374903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.374863 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9"] Oct 31 03:37:48.385555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.384388 5520 cpu_manager.go:414] "containerMap: removing container" podUID="94cd2b30-4669-4778-bf32-43e9cc4fc1de" containerName="payload-pre-processing" Oct 31 03:37:48.385555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.384430 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94cd2b30-4669-4778-bf32-43e9cc4fc1de" containerName="payload-pre-processing" Oct 31 03:37:48.385555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.384642 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="94cd2b30-4669-4778-bf32-43e9cc4fc1de" containerName="payload-pre-processing" Oct 31 03:37:48.389326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.389141 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.399552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.399524 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9"] Oct 31 03:37:48.399886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.399864 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8385c5-openshift-default-dockercfg-vs5dp\"" Oct 31 03:37:48.451001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.450956 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451124 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451192 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qv42\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-kube-api-access-8qv42\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451258 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451359 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451425 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451544 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451585 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.451796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.451781 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.627288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.598220 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.714463 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.714696 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.714794 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.714866 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.714914 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qv42\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-kube-api-access-8qv42\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.714941 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.714982 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.715044 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.715489 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.630748 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.715737 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.750231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.716342 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.787294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.785966 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" event={"ID":"fdb96d9d-566b-4d00-9c59-857a2d53552f","Type":"ContainerStarted","Data":"efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203"} Oct 31 03:37:48.787294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.786036 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" event={"ID":"fdb96d9d-566b-4d00-9c59-857a2d53552f","Type":"ContainerStarted","Data":"15d828ba6a4304b2f5be130293adfbf8c89d6b5ca1e0d184af160ab5f9cc2a6c"} Oct 31 03:37:48.787294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.787217 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:37:48.806291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.796557 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.806291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.803734 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" event={"ID":"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7","Type":"ContainerStarted","Data":"8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5"} Oct 31 03:37:48.806291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.803791 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:48.806291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.803814 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" event={"ID":"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7","Type":"ContainerStarted","Data":"5fd7465a066b5d4c7306d6cd8316f09ab8957afa41a6a8207f160b95fadce318"} Oct 31 03:37:48.829239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.810992 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" podStartSLOduration=2.810974404 podStartE2EDuration="2.810974404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:48.810584461 +0000 UTC m=+1293.005107594" watchObservedRunningTime="2026-10-31 03:37:48.810974404 +0000 UTC m=+1293.005497526" Oct 31 03:37:48.895117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.895029 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.910351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.910308 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.941553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.941522 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.952486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.950934 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk"] Oct 31 03:37:48.962871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.961962 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qv42\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-kube-api-access-8qv42\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:48.964351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.964331 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:48.981655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.981624 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d04aad-dockercfg-z6q7j\"" Oct 31 03:37:48.994149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:48.982081 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d04aad\"" Oct 31 03:37:49.011974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.011940 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk"] Oct 31 03:37:49.082834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.081863 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" podStartSLOduration=2.0818436 podStartE2EDuration="2.0818436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:48.841313414 +0000 UTC m=+1293.035836560" watchObservedRunningTime="2026-10-31 03:37:49.0818436 +0000 UTC m=+1293.276366717" Oct 31 03:37:49.095620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.094377 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:49.221301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.119483 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d79dd-f7ba-478f-9bb8-701683e36f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:49.221301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.177439 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpz8\" (UniqueName: \"kubernetes.io/projected/089d79dd-f7ba-478f-9bb8-701683e36f62-kube-api-access-wlpz8\") pod \"payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:49.284486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.284444 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d79dd-f7ba-478f-9bb8-701683e36f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:49.284689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.284619 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpz8\" (UniqueName: \"kubernetes.io/projected/089d79dd-f7ba-478f-9bb8-701683e36f62-kube-api-access-wlpz8\") pod \"payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:49.317243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.285677 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d79dd-f7ba-478f-9bb8-701683e36f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:49.369408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.367355 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh"] Oct 31 03:37:49.449317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.435497 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:49.453245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.450037 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpz8\" (UniqueName: \"kubernetes.io/projected/089d79dd-f7ba-478f-9bb8-701683e36f62-kube-api-access-wlpz8\") pod \"payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:49.453245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.452084 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh"] Oct 31 03:37:49.472271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.471434 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:49.490754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.474893 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mznw\" (UniqueName: \"kubernetes.io/projected/7e1bca48-0aeb-42b6-b774-240be3e897e9-kube-api-access-7mznw\") pod \"payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:49.490754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.475128 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1bca48-0aeb-42b6-b774-240be3e897e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:49.577047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.577000 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1bca48-0aeb-42b6-b774-240be3e897e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:49.577605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.577580 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mznw\" (UniqueName: \"kubernetes.io/projected/7e1bca48-0aeb-42b6-b774-240be3e897e9-kube-api-access-7mznw\") pod \"payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:49.579277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.579236 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1bca48-0aeb-42b6-b774-240be3e897e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:49.589537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.589506 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mznw\" (UniqueName: \"kubernetes.io/projected/7e1bca48-0aeb-42b6-b774-240be3e897e9-kube-api-access-7mznw\") pod \"payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:49.856468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.856433 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9"] Oct 31 03:37:49.859825 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:49.859786 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf55357_a8e7_49ed_87ba_99aa5e09b6d3.slice/crio-a945ce27efeecb25355b5abfca8cf64a78156a142416f0cebb1b859159b5fa16 WatchSource:0}: Error finding container a945ce27efeecb25355b5abfca8cf64a78156a142416f0cebb1b859159b5fa16: Status 404 returned error can't find the container with id a945ce27efeecb25355b5abfca8cf64a78156a142416f0cebb1b859159b5fa16 Oct 31 03:37:49.863851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.863813 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:37:49.863967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.863900 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:37:49.863967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.863945 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:37:49.911454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.910579 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk"] Oct 31 03:37:49.920305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:49.917415 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:50.324821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.324792 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh"] Oct 31 03:37:50.328677 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:50.328652 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1bca48_0aeb_42b6_b774_240be3e897e9.slice/crio-969235c7d05d7f8292045744f6edf433d6f66aca4affcc050b31362843b5da4b WatchSource:0}: Error finding container 969235c7d05d7f8292045744f6edf433d6f66aca4affcc050b31362843b5da4b: Status 404 returned error can't find the container with id 969235c7d05d7f8292045744f6edf433d6f66aca4affcc050b31362843b5da4b Oct 31 03:37:50.821501 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.821457 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" event={"ID":"089d79dd-f7ba-478f-9bb8-701683e36f62","Type":"ContainerStarted","Data":"eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97"} Oct 31 03:37:50.821683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.821560 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:37:50.821683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.821585 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" event={"ID":"089d79dd-f7ba-478f-9bb8-701683e36f62","Type":"ContainerStarted","Data":"16ece96c6471fc0520bfae0c99b85bdfd994deccaf459a689f4bcea387c7a695"} Oct 31 03:37:50.823239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.823193 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" event={"ID":"7e1bca48-0aeb-42b6-b774-240be3e897e9","Type":"ContainerStarted","Data":"e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617"} Oct 31 03:37:50.823363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.823253 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" event={"ID":"7e1bca48-0aeb-42b6-b774-240be3e897e9","Type":"ContainerStarted","Data":"969235c7d05d7f8292045744f6edf433d6f66aca4affcc050b31362843b5da4b"} Oct 31 03:37:50.823363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.823273 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:37:50.825268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.825246 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" event={"ID":"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3","Type":"ContainerStarted","Data":"6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253"} Oct 31 03:37:50.825354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.825289 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" event={"ID":"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3","Type":"ContainerStarted","Data":"a945ce27efeecb25355b5abfca8cf64a78156a142416f0cebb1b859159b5fa16"} Oct 31 03:37:50.848627 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.848571 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" podStartSLOduration=2.848554027 podStartE2EDuration="2.848554027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:50.845726793 +0000 UTC m=+1295.040249939" watchObservedRunningTime="2026-10-31 03:37:50.848554027 +0000 UTC m=+1295.043077150" Oct 31 03:37:50.873125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.873074 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" podStartSLOduration=2.873060391 podStartE2EDuration="2.873060391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:50.868512621 +0000 UTC m=+1295.063035741" watchObservedRunningTime="2026-10-31 03:37:50.873060391 +0000 UTC m=+1295.067583511" Oct 31 03:37:50.904870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:50.904819 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podStartSLOduration=2.904803765 podStartE2EDuration="2.904803765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:50.903276745 +0000 UTC m=+1295.097799879" watchObservedRunningTime="2026-10-31 03:37:50.904803765 +0000 UTC m=+1295.099326886" Oct 31 03:37:51.150228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:51.147368 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:51.189952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:51.189551 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:51.833949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:51.833916 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:51.834863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:51.834838 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:37:54.483955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.483917 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m"] Oct 31 03:37:54.489834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.489796 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.499364 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.499332 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m"] Oct 31 03:37:54.499531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.499366 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8385c5-dockercfg-2nwkt\"" Oct 31 03:37:54.499616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.499597 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8385c5\"" Oct 31 03:37:54.525253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.524048 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.525253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.524325 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vx7\" (UniqueName: \"kubernetes.io/projected/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-kube-api-access-94vx7\") pod \"payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.526653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.526634 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm"] Oct 31 03:37:54.531242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.531223 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:54.562421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.560679 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm"] Oct 31 03:37:54.695227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.677281 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ca905-81e4-439b-ac0b-facf39d3a7db-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:54.695860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.695823 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmml\" (UniqueName: \"kubernetes.io/projected/7e1ca905-81e4-439b-ac0b-facf39d3a7db-kube-api-access-mkmml\") pod \"payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:54.701226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.696035 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.701507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.701484 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vx7\" (UniqueName: \"kubernetes.io/projected/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-kube-api-access-94vx7\") pod \"payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.717228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.700065 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.728234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.724868 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vx7\" (UniqueName: \"kubernetes.io/projected/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-kube-api-access-94vx7\") pod \"payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.822276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.818041 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ca905-81e4-439b-ac0b-facf39d3a7db-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:54.822276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.818302 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmml\" (UniqueName: \"kubernetes.io/projected/7e1ca905-81e4-439b-ac0b-facf39d3a7db-kube-api-access-mkmml\") pod \"payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:54.822276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.819618 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ca905-81e4-439b-ac0b-facf39d3a7db-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:54.824610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.823424 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:54.845773 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.833424 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmml\" (UniqueName: \"kubernetes.io/projected/7e1ca905-81e4-439b-ac0b-facf39d3a7db-kube-api-access-mkmml\") pod \"payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:54.885547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:54.885509 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:55.284287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.284257 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m"] Oct 31 03:37:55.285327 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:55.285296 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa34fb9_c71c_4a6c_b5f5_5df5f4e7a092.slice/crio-32aee9958bd7af60470700d62c4e416f059d771c8c00e6767184f53666281911 WatchSource:0}: Error finding container 32aee9958bd7af60470700d62c4e416f059d771c8c00e6767184f53666281911: Status 404 returned error can't find the container with id 32aee9958bd7af60470700d62c4e416f059d771c8c00e6767184f53666281911 Oct 31 03:37:55.295445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.295418 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm"] Oct 31 03:37:55.295753 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:37:55.295716 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1ca905_81e4_439b_ac0b_facf39d3a7db.slice/crio-6f0edaf51960d87fad6729b5ae45be15e0eef89a3869aea664ffaaaf899db989 WatchSource:0}: Error finding container 6f0edaf51960d87fad6729b5ae45be15e0eef89a3869aea664ffaaaf899db989: Status 404 returned error can't find the container with id 6f0edaf51960d87fad6729b5ae45be15e0eef89a3869aea664ffaaaf899db989 Oct 31 03:37:55.948693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.948662 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" event={"ID":"7e1ca905-81e4-439b-ac0b-facf39d3a7db","Type":"ContainerStarted","Data":"1e0c9f30ab12cfbd70a79b7e474ec087d04fee877b3c86bcf944b2b6f1c358ca"} Oct 31 03:37:55.949189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.948723 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" event={"ID":"7e1ca905-81e4-439b-ac0b-facf39d3a7db","Type":"ContainerStarted","Data":"6f0edaf51960d87fad6729b5ae45be15e0eef89a3869aea664ffaaaf899db989"} Oct 31 03:37:55.949189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.948753 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:37:55.950626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.950600 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" event={"ID":"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092","Type":"ContainerStarted","Data":"30dc7b719af2de123c3763ca1af79037d5d5730a951f5898ca4b555d0177a735"} Oct 31 03:37:55.950727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.950639 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" event={"ID":"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092","Type":"ContainerStarted","Data":"32aee9958bd7af60470700d62c4e416f059d771c8c00e6767184f53666281911"} Oct 31 03:37:55.950727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.950665 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:37:55.969407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.969331 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" podStartSLOduration=1.969314121 podStartE2EDuration="1.969314121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:55.966186063 +0000 UTC m=+1300.160709221" watchObservedRunningTime="2026-10-31 03:37:55.969314121 +0000 UTC m=+1300.163837244" Oct 31 03:37:55.987125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:55.987062 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" podStartSLOduration=1.987017964 podStartE2EDuration="1.987017964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:55.986133227 +0000 UTC m=+1300.180656347" watchObservedRunningTime="2026-10-31 03:37:55.987017964 +0000 UTC m=+1300.181541085" Oct 31 03:37:56.916617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.916587 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz"] Oct 31 03:37:56.923680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.922746 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.932014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.930927 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-34dbeb-openshift-default-dockercfg-lw5vl\"" Oct 31 03:37:56.936345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.936319 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz"] Oct 31 03:37:56.973865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.973832 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-envoy\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.973904 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-certs\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.973992 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-socket\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.974439 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-token\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.974532 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istiod-ca-cert\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.974613 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvlr\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-kube-api-access-xbvlr\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.974741 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-podinfo\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.974802 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-data\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:56.974891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:56.974863 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-credential-socket\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148563 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-token\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148610 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istiod-ca-cert\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148644 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvlr\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-kube-api-access-xbvlr\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148745 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-podinfo\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148791 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-data\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148832 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-credential-socket\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148933 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-envoy\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.148966 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-certs\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.149044 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-socket\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.152236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.149590 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-socket\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.214312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.198787 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-data\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.214312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.200799 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-certs\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.214312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.201029 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-credential-socket\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.284397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.279506 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istiod-ca-cert\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.290330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.280456 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-envoy\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.322229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.303952 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-podinfo\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.366229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.340973 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-token\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.366229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.341700 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvlr\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-kube-api-access-xbvlr\") pod \"e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.553750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.553660 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:57.690458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.690419 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8"] Oct 31 03:37:57.740259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.738735 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8"] Oct 31 03:37:57.740259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.738861 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.772536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.770706 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-adb7b7-serving-cert\"" Oct 31 03:37:57.816182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.816098 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-metrics-certs\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.816547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.816524 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-openshift-service-ca\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.816889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.816869 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-maas-api-tls\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.817028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.817013 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpbw\" (UniqueName: \"kubernetes.io/projected/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-kube-api-access-xgpbw\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.935762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.935718 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-maas-api-tls\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.936034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.936013 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpbw\" (UniqueName: \"kubernetes.io/projected/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-kube-api-access-xgpbw\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.936447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.936427 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-metrics-certs\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.936935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.936676 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-openshift-service-ca\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.938291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.938024 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-openshift-service-ca\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.975396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.967861 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-maas-api-tls\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.975396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.972842 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-metrics-certs\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:57.985239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:57.978024 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpbw\" (UniqueName: \"kubernetes.io/projected/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-kube-api-access-xgpbw\") pod \"maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:58.093292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:58.090517 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:37:58.204833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:58.202235 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz"] Oct 31 03:37:58.227769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:58.226723 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:37:58.227769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:58.226840 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:37:58.227769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:58.226883 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:37:58.516226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:58.516172 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8"] Oct 31 03:37:59.093230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.092921 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw"] Oct 31 03:37:59.118228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.099150 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.118228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.108038 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-555bd6-serving-cert\"" Oct 31 03:37:59.118228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.111987 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw"] Oct 31 03:37:59.142578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.142382 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" event={"ID":"44a96e65-fb7f-4bc9-88f7-2bf66a97d460","Type":"ContainerStarted","Data":"482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90"} Oct 31 03:37:59.142578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.142447 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" event={"ID":"44a96e65-fb7f-4bc9-88f7-2bf66a97d460","Type":"ContainerStarted","Data":"7be273c7080a3e7cf2558cb76c0da6059a7ee4864d571aa7047e91b4ac59d1ea"} Oct 31 03:37:59.146263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.145051 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" event={"ID":"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d","Type":"ContainerStarted","Data":"942aaa732e8e67546c30ed0a655383ebe8f501fdec8cce6ecc4d1ce766be7805"} Oct 31 03:37:59.165494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.165433 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podStartSLOduration=3.165414344 podStartE2EDuration="3.165414344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:59.161616268 +0000 UTC m=+1303.356139478" watchObservedRunningTime="2026-10-31 03:37:59.165414344 +0000 UTC m=+1303.359937466" Oct 31 03:37:59.220134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.220101 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-metrics-certs\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.220315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.220151 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1731d4-7cce-41f3-945f-ffded1fad0a8-openshift-service-ca\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.220426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.220407 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-maas-api-tls\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.220478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.220443 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhvp\" (UniqueName: \"kubernetes.io/projected/fd1731d4-7cce-41f3-945f-ffded1fad0a8-kube-api-access-bfhvp\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.324401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.324094 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-maas-api-tls\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.324401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.324152 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhvp\" (UniqueName: \"kubernetes.io/projected/fd1731d4-7cce-41f3-945f-ffded1fad0a8-kube-api-access-bfhvp\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.341280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.324431 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-metrics-certs\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.341630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.341604 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1731d4-7cce-41f3-945f-ffded1fad0a8-openshift-service-ca\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.348275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.342622 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1731d4-7cce-41f3-945f-ffded1fad0a8-openshift-service-ca\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.397939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.397902 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-maas-api-tls\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.432815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.431278 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-metrics-certs\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.443229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.440888 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhvp\" (UniqueName: \"kubernetes.io/projected/fd1731d4-7cce-41f3-945f-ffded1fad0a8-kube-api-access-bfhvp\") pod \"maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.554006 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.553960 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:59.560752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.560720 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:37:59.733055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.733014 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:37:59.856231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.849338 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:37:59.856231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.851404 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:37:59.856231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:37:59.852473 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:38:00.387224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:00.387130 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:38:00.421262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:00.403227 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" event={"ID":"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d","Type":"ContainerStarted","Data":"5f7b00b85aa0bbf1e282bc7af02986d72d8c7670cf943ef43863417db26fadd3"} Oct 31 03:38:00.421262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:00.403285 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:38:00.421262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:00.403350 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:38:00.425979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:00.425920 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" podStartSLOduration=2.910557054 podStartE2EDuration="3.425900218s" totalImagesPullingTime="515.343164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:00.420242846 +0000 UTC m=+1304.614765970" watchObservedRunningTime="2026-10-31 03:38:00.425900218 +0000 UTC m=+1304.620423340" Oct 31 03:38:00.501876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:00.501398 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw"] Oct 31 03:38:00.859279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:00.858405 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:38:01.488988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:01.488393 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" event={"ID":"fd1731d4-7cce-41f3-945f-ffded1fad0a8","Type":"ContainerStarted","Data":"bce39fba7e75d1bf31281466a213de8196338eb9654a8c7c87de846bead04cb2"} Oct 31 03:38:01.871227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:01.866407 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:38:01.871227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:01.868879 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:38:02.575231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:02.569467 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" event={"ID":"fd1731d4-7cce-41f3-945f-ffded1fad0a8","Type":"ContainerStarted","Data":"f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b"} Oct 31 03:38:02.575231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:02.569552 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:38:02.638320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:02.638271 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" podStartSLOduration=2.86978586 podStartE2EDuration="3.638254269s" totalImagesPullingTime="768.468409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:02.634612508 +0000 UTC m=+1306.829135629" watchObservedRunningTime="2026-10-31 03:38:02.638254269 +0000 UTC m=+1306.832777637" Oct 31 03:38:06.281873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:06.281834 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656"] Oct 31 03:38:06.282405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:06.282088 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" podUID="311a29cd-0635-4cd6-8ed7-ef31f7f80134" containerName="maas-api" containerID="cri-o://2d5c9f2ad0c7778f651b35a3c020bb71bc9374fd63dc35c7162d983703f05a7b" gracePeriod=30 Oct 31 03:38:06.849703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:06.817399 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:38:07.163395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.129871 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:38:07.163395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.159344 5520 generic.go:356] "Generic (PLEG): container finished" podID="311a29cd-0635-4cd6-8ed7-ef31f7f80134" containerID="2d5c9f2ad0c7778f651b35a3c020bb71bc9374fd63dc35c7162d983703f05a7b" exitCode=0 Oct 31 03:38:07.163395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.159630 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" event={"ID":"311a29cd-0635-4cd6-8ed7-ef31f7f80134","Type":"ContainerDied","Data":"2d5c9f2ad0c7778f651b35a3c020bb71bc9374fd63dc35c7162d983703f05a7b"} Oct 31 03:38:07.195981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.195947 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:38:07.622424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.622400 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:38:07.748188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.748149 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311a29cd-0635-4cd6-8ed7-ef31f7f80134-kube-api-access-7rtf4\" (UniqueName: \"kubernetes.io/projected/311a29cd-0635-4cd6-8ed7-ef31f7f80134-kube-api-access-7rtf4\") pod \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " Oct 31 03:38:07.748374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.748238 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-metrics-certs\") pod \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " Oct 31 03:38:07.748457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.748421 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311a29cd-0635-4cd6-8ed7-ef31f7f80134-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311a29cd-0635-4cd6-8ed7-ef31f7f80134-openshift-service-ca\") pod \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " Oct 31 03:38:07.748570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.748556 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-maas-api-tls\") pod \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\" (UID: \"311a29cd-0635-4cd6-8ed7-ef31f7f80134\") " Oct 31 03:38:07.748790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.748764 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311a29cd-0635-4cd6-8ed7-ef31f7f80134-openshift-service-ca" pod "311a29cd-0635-4cd6-8ed7-ef31f7f80134" (UID: "311a29cd-0635-4cd6-8ed7-ef31f7f80134"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:07.749247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.749225 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/311a29cd-0635-4cd6-8ed7-ef31f7f80134-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:07.751920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.751874 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-maas-api-tls" pod "311a29cd-0635-4cd6-8ed7-ef31f7f80134" (UID: "311a29cd-0635-4cd6-8ed7-ef31f7f80134"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:07.753903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.753876 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-metrics-certs" pod "311a29cd-0635-4cd6-8ed7-ef31f7f80134" (UID: "311a29cd-0635-4cd6-8ed7-ef31f7f80134"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:07.759608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.758689 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311a29cd-0635-4cd6-8ed7-ef31f7f80134-kube-api-access-7rtf4" pod "311a29cd-0635-4cd6-8ed7-ef31f7f80134" (UID: "311a29cd-0635-4cd6-8ed7-ef31f7f80134"). InnerVolumeSpecName "kube-api-access-7rtf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:07.774787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.774758 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb"] Oct 31 03:38:07.775755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.775734 5520 cpu_manager.go:414] "containerMap: removing container" podUID="311a29cd-0635-4cd6-8ed7-ef31f7f80134" containerName="maas-api" Oct 31 03:38:07.775853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.775754 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311a29cd-0635-4cd6-8ed7-ef31f7f80134" containerName="maas-api" Oct 31 03:38:07.775920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.775882 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="311a29cd-0635-4cd6-8ed7-ef31f7f80134" containerName="maas-api" Oct 31 03:38:07.779893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.779876 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:07.790147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.790122 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb"] Oct 31 03:38:07.790421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.790406 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-74c760\"" Oct 31 03:38:07.802035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.802010 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-74c760-dockercfg-rzbtp\"" Oct 31 03:38:07.850696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.850615 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74235b76-3bed-49e7-98ae-a300f7be917f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:07.850836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.850699 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgtp\" (UniqueName: \"kubernetes.io/projected/74235b76-3bed-49e7-98ae-a300f7be917f-kube-api-access-4fgtp\") pod \"payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:07.851003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.850983 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtf4\" (UniqueName: \"kubernetes.io/projected/311a29cd-0635-4cd6-8ed7-ef31f7f80134-kube-api-access-7rtf4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:07.851041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.851007 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:07.851041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.851018 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/311a29cd-0635-4cd6-8ed7-ef31f7f80134-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:07.951891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.951863 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74235b76-3bed-49e7-98ae-a300f7be917f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:07.952092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.951987 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgtp\" (UniqueName: \"kubernetes.io/projected/74235b76-3bed-49e7-98ae-a300f7be917f-kube-api-access-4fgtp\") pod \"payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:07.952693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.952672 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74235b76-3bed-49e7-98ae-a300f7be917f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:07.960228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:07.960182 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgtp\" (UniqueName: \"kubernetes.io/projected/74235b76-3bed-49e7-98ae-a300f7be917f-kube-api-access-4fgtp\") pod \"payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:08.121240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.119917 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:08.243216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.241862 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" event={"ID":"311a29cd-0635-4cd6-8ed7-ef31f7f80134","Type":"ContainerDied","Data":"6ce47b66116193cee1ea0e283dff53311ba9064e228ef91fb0e08fc3959b7e6a"} Oct 31 03:38:08.243216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.241933 5520 scope.go:122] "RemoveContainer" containerID="2d5c9f2ad0c7778f651b35a3c020bb71bc9374fd63dc35c7162d983703f05a7b" Oct 31 03:38:08.243216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.242103 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656" Oct 31 03:38:08.307428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.307351 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656"] Oct 31 03:38:08.312169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.312138 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fc226c-5b4849848f-fx656"] Oct 31 03:38:08.582038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.582008 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:38:08.635172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:08.635139 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb"] Oct 31 03:38:09.321906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:09.320480 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" event={"ID":"74235b76-3bed-49e7-98ae-a300f7be917f","Type":"ContainerStarted","Data":"386191c08999d1a33f76eab1bbe5ff8deb414396784555accb52179c1b03d2f2"} Oct 31 03:38:09.321906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:09.320556 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" event={"ID":"74235b76-3bed-49e7-98ae-a300f7be917f","Type":"ContainerStarted","Data":"696b0f37f583cbbc6096f339939cbd885f5daf765da14fafda4fcbe830e03361"} Oct 31 03:38:09.321906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:09.321850 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:09.362870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:09.362810 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" podStartSLOduration=2.362790203 podStartE2EDuration="2.362790203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:09.356805711 +0000 UTC m=+1313.551328832" watchObservedRunningTime="2026-10-31 03:38:09.362790203 +0000 UTC m=+1313.557313324" Oct 31 03:38:09.431569 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:38:09.431518 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:38:09.930573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:09.930541 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311a29cd-0635-4cd6-8ed7-ef31f7f80134" path="/var/lib/kubelet/pods/311a29cd-0635-4cd6-8ed7-ef31f7f80134/volumes" Oct 31 03:38:19.501074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.501038 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq"] Oct 31 03:38:19.522626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.522588 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq"] Oct 31 03:38:19.522795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.522742 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.526717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.526687 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-34dbeb-serving-cert\"" Oct 31 03:38:19.683226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.679595 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/62e55949-1870-4a62-8a9a-cc22317fb556-kube-api-access-5b5fr\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.683226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.679754 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-maas-api-tls\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.683226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.679843 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e55949-1870-4a62-8a9a-cc22317fb556-openshift-service-ca\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.683226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.680022 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-metrics-certs\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.782288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.781365 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/62e55949-1870-4a62-8a9a-cc22317fb556-kube-api-access-5b5fr\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.782288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.781510 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-maas-api-tls\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.782288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.781593 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e55949-1870-4a62-8a9a-cc22317fb556-openshift-service-ca\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.782288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.781752 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-metrics-certs\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.797004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.795107 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e55949-1870-4a62-8a9a-cc22317fb556-openshift-service-ca\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.797784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.797757 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-metrics-certs\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.798862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.798832 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-maas-api-tls\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.819968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.817752 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/62e55949-1870-4a62-8a9a-cc22317fb556-kube-api-access-5b5fr\") pod \"maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:19.841905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:19.841878 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:20.253870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:20.253841 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq"] Oct 31 03:38:20.556263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:20.553470 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" event={"ID":"62e55949-1870-4a62-8a9a-cc22317fb556","Type":"ContainerStarted","Data":"78f2d8bd41442ba0cd0eef7b4657cd844ee12359340b0e3ffc74d52e99a63eaa"} Oct 31 03:38:21.374229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:21.373446 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:38:21.621227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:21.620336 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" event={"ID":"62e55949-1870-4a62-8a9a-cc22317fb556","Type":"ContainerStarted","Data":"5d969d565dc1a0a0a48dd5d8bddc489ea4a5cf0751bdf78c4ccce7bcd88a0589"} Oct 31 03:38:21.621788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:21.621589 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:21.650915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:21.650349 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" podStartSLOduration=2.205510411 podStartE2EDuration="2.650324445s" totalImagesPullingTime="444.814034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:21.645729617 +0000 UTC m=+1325.840252739" watchObservedRunningTime="2026-10-31 03:38:21.650324445 +0000 UTC m=+1325.844847570" Oct 31 03:38:23.386734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.386699 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch"] Oct 31 03:38:23.399547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.399514 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.416914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.416883 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-74c760-serving-cert\"" Oct 31 03:38:23.423460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.423431 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch"] Oct 31 03:38:23.450983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.450946 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-maas-api-tls\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.451191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.451017 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-openshift-service-ca\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.451191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.451113 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-metrics-certs\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.451191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.451137 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktgv\" (UniqueName: \"kubernetes.io/projected/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-kube-api-access-5ktgv\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.556558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.556507 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-maas-api-tls\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.563024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.562982 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-openshift-service-ca\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.563528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.563496 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-metrics-certs\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.563748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.563716 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktgv\" (UniqueName: \"kubernetes.io/projected/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-kube-api-access-5ktgv\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.564608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.562822 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-maas-api-tls\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.565540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.565510 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-openshift-service-ca\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.596771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.596741 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-metrics-certs\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.612226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.597105 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktgv\" (UniqueName: \"kubernetes.io/projected/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-kube-api-access-5ktgv\") pod \"maas-api-e2e-api-a-74c760-86dbfb544b-tdtch\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:23.734961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:23.734926 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:24.450358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:24.424057 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch"] Oct 31 03:38:24.606344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:24.606315 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x"] Oct 31 03:38:24.611636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:24.611606 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx"] Oct 31 03:38:24.718820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:24.718735 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" event={"ID":"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55","Type":"ContainerStarted","Data":"3ec0ce440390b91e500954ad61eaf0f242983ff878cdc77fdf3b3c078e008658"} Oct 31 03:38:25.729090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:25.729040 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" event={"ID":"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55","Type":"ContainerStarted","Data":"f0449984b5afdc459945161d7b3d281a0b8b1972d1966b12c8afc9be80a5543d"} Oct 31 03:38:25.729580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:25.729130 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:25.750964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:25.750898 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" podStartSLOduration=2.237651763 podStartE2EDuration="2.750879346s" totalImagesPullingTime="513.227583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:25.747400898 +0000 UTC m=+1329.941924021" watchObservedRunningTime="2026-10-31 03:38:25.750879346 +0000 UTC m=+1329.945402469" Oct 31 03:38:27.722053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:27.722012 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr"] Oct 31 03:38:27.728146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:27.728121 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:27.738509 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:27.738476 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr"] Oct 31 03:38:27.796597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:27.796562 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-619d46-dockercfg-5w94j\"" Oct 31 03:38:27.796974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:27.796916 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-619d46\"" Oct 31 03:38:27.814415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:27.813158 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300883b1-3b94-4ff2-88bd-668000fa70f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:27.814415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:27.813461 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbb5\" (UniqueName: \"kubernetes.io/projected/300883b1-3b94-4ff2-88bd-668000fa70f7-kube-api-access-8hbb5\") pod \"payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:28.022367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:28.022275 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbb5\" (UniqueName: \"kubernetes.io/projected/300883b1-3b94-4ff2-88bd-668000fa70f7-kube-api-access-8hbb5\") pod \"payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:28.023613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:28.023586 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300883b1-3b94-4ff2-88bd-668000fa70f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:28.024903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:28.024880 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300883b1-3b94-4ff2-88bd-668000fa70f7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:28.093005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:28.092971 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbb5\" (UniqueName: \"kubernetes.io/projected/300883b1-3b94-4ff2-88bd-668000fa70f7-kube-api-access-8hbb5\") pod \"payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:28.357520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:28.357437 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:28.760076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:28.760035 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr"] Oct 31 03:38:28.811215 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:38:28.810096 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300883b1_3b94_4ff2_88bd_668000fa70f7.slice/crio-bab63399a45f7ac19cc23fa50ce5eb4581cb6051725130ee8eba93bee4bae20d WatchSource:0}: Error finding container bab63399a45f7ac19cc23fa50ce5eb4581cb6051725130ee8eba93bee4bae20d: Status 404 returned error can't find the container with id bab63399a45f7ac19cc23fa50ce5eb4581cb6051725130ee8eba93bee4bae20d Oct 31 03:38:28.817606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:28.816135 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:38:29.141476 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.141439 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" event={"ID":"300883b1-3b94-4ff2-88bd-668000fa70f7","Type":"ContainerStarted","Data":"bab63399a45f7ac19cc23fa50ce5eb4581cb6051725130ee8eba93bee4bae20d"} Oct 31 03:38:29.442675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.442634 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8"] Oct 31 03:38:29.447853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.447827 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:29.455303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.455274 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8361df\"" Oct 31 03:38:29.458025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.457978 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8"] Oct 31 03:38:29.487590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.487554 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f"] Oct 31 03:38:29.492472 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.492448 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:29.500853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.500827 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f"] Oct 31 03:38:29.513044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.513012 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8361df-dockercfg-7wdhx\"" Oct 31 03:38:29.629230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.628338 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" podUID="7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" containerName="payload-processing" containerID="cri-o://e64ee0379bfd6c2850dea66d02ebf4421c197ada27f739f19bbc743a511a63fb" gracePeriod=25 Oct 31 03:38:29.659229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.657095 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff74a954-22a6-4353-a2e5-0ac91d6137a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:29.659229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.657158 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vtn\" (UniqueName: \"kubernetes.io/projected/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-kube-api-access-b5vtn\") pod \"payload-processing-e2e-api-b-8361df-79785f96df-fcf5f\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:29.659229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.657236 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8361df-79785f96df-fcf5f\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:29.659229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.657346 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw6s2\" (UniqueName: \"kubernetes.io/projected/ff74a954-22a6-4353-a2e5-0ac91d6137a0-kube-api-access-rw6s2\") pod \"payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:29.703274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.696251 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" podUID="00d47495-d197-4a35-a556-fdd27b25658a" containerName="payload-pre-processing" containerID="cri-o://c69ff1f7dc09f7a1a1ea40d2cceb5039f5a5200e12d168608c239583106e4a9a" gracePeriod=25 Oct 31 03:38:29.771228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.768245 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff74a954-22a6-4353-a2e5-0ac91d6137a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:29.796616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.796471 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff74a954-22a6-4353-a2e5-0ac91d6137a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:29.796616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.796544 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vtn\" (UniqueName: \"kubernetes.io/projected/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-kube-api-access-b5vtn\") pod \"payload-processing-e2e-api-b-8361df-79785f96df-fcf5f\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:29.796616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.796592 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8361df-79785f96df-fcf5f\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:29.796862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.796692 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw6s2\" (UniqueName: \"kubernetes.io/projected/ff74a954-22a6-4353-a2e5-0ac91d6137a0-kube-api-access-rw6s2\") pod \"payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:29.814846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.814613 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vtn\" (UniqueName: \"kubernetes.io/projected/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-kube-api-access-b5vtn\") pod \"payload-processing-e2e-api-b-8361df-79785f96df-fcf5f\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:29.845704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.844935 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8361df-79785f96df-fcf5f\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:29.861307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:29.860928 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw6s2\" (UniqueName: \"kubernetes.io/projected/ff74a954-22a6-4353-a2e5-0ac91d6137a0-kube-api-access-rw6s2\") pod \"payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:30.068670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.068260 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:30.115051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.115014 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:30.461227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.457459 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" event={"ID":"300883b1-3b94-4ff2-88bd-668000fa70f7","Type":"ContainerStarted","Data":"0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a"} Oct 31 03:38:30.461227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.458279 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:30.573581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.571536 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" podStartSLOduration=3.571512373 podStartE2EDuration="3.571512373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:30.549986627 +0000 UTC m=+1334.744509770" watchObservedRunningTime="2026-10-31 03:38:30.571512373 +0000 UTC m=+1334.766035495" Oct 31 03:38:30.593312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.590850 5520 generic.go:356] "Generic (PLEG): container finished" podID="00d47495-d197-4a35-a556-fdd27b25658a" containerID="c69ff1f7dc09f7a1a1ea40d2cceb5039f5a5200e12d168608c239583106e4a9a" exitCode=0 Oct 31 03:38:30.593312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.591048 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" event={"ID":"00d47495-d197-4a35-a556-fdd27b25658a","Type":"ContainerDied","Data":"c69ff1f7dc09f7a1a1ea40d2cceb5039f5a5200e12d168608c239583106e4a9a"} Oct 31 03:38:30.716235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.669559 5520 generic.go:356] "Generic (PLEG): container finished" podID="7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" containerID="e64ee0379bfd6c2850dea66d02ebf4421c197ada27f739f19bbc743a511a63fb" exitCode=0 Oct 31 03:38:30.716235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.669689 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" event={"ID":"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38","Type":"ContainerDied","Data":"e64ee0379bfd6c2850dea66d02ebf4421c197ada27f739f19bbc743a511a63fb"} Oct 31 03:38:30.717331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.717302 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:38:30.875228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.872559 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-plugins-config-volume\") pod \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " Oct 31 03:38:30.875228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.872665 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-kube-api-access-kf84c\" (UniqueName: \"kubernetes.io/projected/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-kube-api-access-kf84c\") pod \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\" (UID: \"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38\") " Oct 31 03:38:30.882731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.882471 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-kube-api-access-kf84c" pod "7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" (UID: "7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38"). InnerVolumeSpecName "kube-api-access-kf84c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:30.883226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.883092 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-plugins-config-volume" pod "7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" (UID: "7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:30.945224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.942708 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:38:31.004467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.989410 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d47495-d197-4a35-a556-fdd27b25658a-kube-api-access-j6z56\" (UniqueName: \"kubernetes.io/projected/00d47495-d197-4a35-a556-fdd27b25658a-kube-api-access-j6z56\") pod \"00d47495-d197-4a35-a556-fdd27b25658a\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " Oct 31 03:38:31.004467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.989595 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d47495-d197-4a35-a556-fdd27b25658a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d47495-d197-4a35-a556-fdd27b25658a-plugins-config-volume\") pod \"00d47495-d197-4a35-a556-fdd27b25658a\" (UID: \"00d47495-d197-4a35-a556-fdd27b25658a\") " Oct 31 03:38:31.004467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.990152 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:31.004467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.990174 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf84c\" (UniqueName: \"kubernetes.io/projected/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38-kube-api-access-kf84c\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:31.004467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.993301 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d47495-d197-4a35-a556-fdd27b25658a-plugins-config-volume" pod "00d47495-d197-4a35-a556-fdd27b25658a" (UID: "00d47495-d197-4a35-a556-fdd27b25658a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:31.004467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:30.993589 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d47495-d197-4a35-a556-fdd27b25658a-kube-api-access-j6z56" pod "00d47495-d197-4a35-a556-fdd27b25658a" (UID: "00d47495-d197-4a35-a556-fdd27b25658a"). InnerVolumeSpecName "kube-api-access-j6z56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:31.093662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.093626 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d47495-d197-4a35-a556-fdd27b25658a-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:31.093840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.093663 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6z56\" (UniqueName: \"kubernetes.io/projected/00d47495-d197-4a35-a556-fdd27b25658a-kube-api-access-j6z56\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:31.219452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.219410 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f"] Oct 31 03:38:31.226216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.226172 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8"] Oct 31 03:38:31.679263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.679230 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" event={"ID":"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8","Type":"ContainerStarted","Data":"d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9"} Oct 31 03:38:31.679453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.679293 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" event={"ID":"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8","Type":"ContainerStarted","Data":"320b6e68db5fa77fc4893626e9d6d693fc02903b38f721d68d207a1bd7cabf8b"} Oct 31 03:38:31.679530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.679468 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:31.681131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.681113 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" Oct 31 03:38:31.681274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.681132 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx" event={"ID":"00d47495-d197-4a35-a556-fdd27b25658a","Type":"ContainerDied","Data":"fe6b2a9507bbb521992ba8008ac20ed6b46a9556c6e0c44cc22448fe08586b3a"} Oct 31 03:38:31.681274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.681185 5520 scope.go:122] "RemoveContainer" containerID="c69ff1f7dc09f7a1a1ea40d2cceb5039f5a5200e12d168608c239583106e4a9a" Oct 31 03:38:31.683454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.683377 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" Oct 31 03:38:31.683454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.683398 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x" event={"ID":"7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38","Type":"ContainerDied","Data":"1cc55af2a56db8c2cf353e318f5286b4abb7df17460183a23cdaf1b218599f3f"} Oct 31 03:38:31.685692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.685659 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" event={"ID":"ff74a954-22a6-4353-a2e5-0ac91d6137a0","Type":"ContainerStarted","Data":"9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c"} Oct 31 03:38:31.685807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.685709 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:31.685807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.685730 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" event={"ID":"ff74a954-22a6-4353-a2e5-0ac91d6137a0","Type":"ContainerStarted","Data":"9a9e8d314b03b1e03149018e9dabc51bc1b1facd90ad5fd017f31c390d362e0d"} Oct 31 03:38:31.696737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.696719 5520 scope.go:122] "RemoveContainer" containerID="e64ee0379bfd6c2850dea66d02ebf4421c197ada27f739f19bbc743a511a63fb" Oct 31 03:38:31.698164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.697907 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" podStartSLOduration=2.697891727 podStartE2EDuration="2.697891727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:31.696914849 +0000 UTC m=+1335.891437991" watchObservedRunningTime="2026-10-31 03:38:31.697891727 +0000 UTC m=+1335.892414848" Oct 31 03:38:31.715519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.715481 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" podStartSLOduration=2.715469384 podStartE2EDuration="2.715469384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:31.71366791 +0000 UTC m=+1335.908191051" watchObservedRunningTime="2026-10-31 03:38:31.715469384 +0000 UTC m=+1335.909992519" Oct 31 03:38:31.728952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.728929 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x"] Oct 31 03:38:31.733051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.733027 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b87769-7f86b8c68c-kbg5x"] Oct 31 03:38:31.739714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.739688 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx"] Oct 31 03:38:31.745104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.745082 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b87769-b7ffc8cf9-xnfcx"] Oct 31 03:38:31.775296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.775276 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:38:31.936576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.936491 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d47495-d197-4a35-a556-fdd27b25658a" path="/var/lib/kubelet/pods/00d47495-d197-4a35-a556-fdd27b25658a/volumes" Oct 31 03:38:31.938099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:31.938074 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" path="/var/lib/kubelet/pods/7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38/volumes" Oct 31 03:38:33.627947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.627914 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk"] Oct 31 03:38:33.628730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.628714 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" containerName="payload-processing" Oct 31 03:38:33.628780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.628731 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" containerName="payload-processing" Oct 31 03:38:33.628780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.628754 5520 cpu_manager.go:414] "containerMap: removing container" podUID="00d47495-d197-4a35-a556-fdd27b25658a" containerName="payload-pre-processing" Oct 31 03:38:33.628780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.628759 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d47495-d197-4a35-a556-fdd27b25658a" containerName="payload-pre-processing" Oct 31 03:38:33.628904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.628895 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d47495-d197-4a35-a556-fdd27b25658a" containerName="payload-pre-processing" Oct 31 03:38:33.628941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.628915 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9a7093-aaee-4d4e-b8d1-1d8f957e1c38" containerName="payload-processing" Oct 31 03:38:33.632457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.632437 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:33.638039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.638015 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b2dc82\"" Oct 31 03:38:33.639611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.639584 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b2dc82-dockercfg-nlfq6\"" Oct 31 03:38:33.641610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.641561 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk"] Oct 31 03:38:33.681246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.679870 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q"] Oct 31 03:38:33.692502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.685850 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q"] Oct 31 03:38:33.692502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.685968 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:33.866101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.864907 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8fg\" (UniqueName: \"kubernetes.io/projected/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-kube-api-access-rk8fg\") pod \"payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:33.866101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.865010 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbpbs\" (UniqueName: \"kubernetes.io/projected/cc412090-34af-455d-b359-890af2e14f43-kube-api-access-tbpbs\") pod \"payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:33.866101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.865053 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:34.122258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:33.865190 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc412090-34af-455d-b359-890af2e14f43-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:34.228948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.228915 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8fg\" (UniqueName: \"kubernetes.io/projected/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-kube-api-access-rk8fg\") pod \"payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:34.230604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.230581 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpbs\" (UniqueName: \"kubernetes.io/projected/cc412090-34af-455d-b359-890af2e14f43-kube-api-access-tbpbs\") pod \"payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:34.230895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.230868 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:34.231238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.231216 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc412090-34af-455d-b359-890af2e14f43-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:34.231866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.231835 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:34.232664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.232544 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc412090-34af-455d-b359-890af2e14f43-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:34.241910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.241863 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8fg\" (UniqueName: \"kubernetes.io/projected/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-kube-api-access-rk8fg\") pod \"payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:34.243971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.243919 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpbs\" (UniqueName: \"kubernetes.io/projected/cc412090-34af-455d-b359-890af2e14f43-kube-api-access-tbpbs\") pod \"payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:34.258160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.258128 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:34.270244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.270212 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:34.631234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.631176 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk"] Oct 31 03:38:34.668504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.668457 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q"] Oct 31 03:38:34.671033 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:38:34.671009 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc412090_34af_455d_b359_890af2e14f43.slice/crio-de1b9d391e583c23ed4a21b75124be7009f9ec061d6282c555f302ddcbcdedc4 WatchSource:0}: Error finding container de1b9d391e583c23ed4a21b75124be7009f9ec061d6282c555f302ddcbcdedc4: Status 404 returned error can't find the container with id de1b9d391e583c23ed4a21b75124be7009f9ec061d6282c555f302ddcbcdedc4 Oct 31 03:38:34.829058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.829024 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" event={"ID":"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b","Type":"ContainerStarted","Data":"fc035a18948804f7fc946592fc1cb1b76bcbcc50e204706af45ed00eb00596a7"} Oct 31 03:38:34.832153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.832118 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" event={"ID":"cc412090-34af-455d-b359-890af2e14f43","Type":"ContainerStarted","Data":"9a4c6560473e28db8b728935ab306f48cf14cd8f734a72d823b9f8a6afed684e"} Oct 31 03:38:34.832326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.832180 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" event={"ID":"cc412090-34af-455d-b359-890af2e14f43","Type":"ContainerStarted","Data":"de1b9d391e583c23ed4a21b75124be7009f9ec061d6282c555f302ddcbcdedc4"} Oct 31 03:38:34.832522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.832502 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:34.853831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:34.853763 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" podStartSLOduration=1.853742527 podStartE2EDuration="1.853742527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:34.852059257 +0000 UTC m=+1339.046582404" watchObservedRunningTime="2026-10-31 03:38:34.853742527 +0000 UTC m=+1339.048265649" Oct 31 03:38:35.844261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:35.844181 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" event={"ID":"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b","Type":"ContainerStarted","Data":"f6b5bfa763e3c426cc7125e888b4a90f0d79eac5eb3ed5a37ef98c305be9912e"} Oct 31 03:38:35.844792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:35.844282 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:35.863169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:35.863121 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" podStartSLOduration=2.863108034 podStartE2EDuration="2.863108034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:35.858841914 +0000 UTC m=+1340.053365036" watchObservedRunningTime="2026-10-31 03:38:35.863108034 +0000 UTC m=+1340.057631154" Oct 31 03:38:39.360026 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:38:39.359982 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:38:39.924252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.924192 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j"] Oct 31 03:38:39.929180 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.929158 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:39.935046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.935023 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6f28b4\"" Oct 31 03:38:39.935391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.935373 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6f28b4-dockercfg-wd7zl\"" Oct 31 03:38:39.937251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.937223 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j"] Oct 31 03:38:39.963821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.963791 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh"] Oct 31 03:38:39.976155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.976120 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:39.980798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:39.980766 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh"] Oct 31 03:38:40.238225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.226534 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:40.247350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.247319 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxc5p\" (UniqueName: \"kubernetes.io/projected/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-kube-api-access-qxc5p\") pod \"payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:40.304229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.294316 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26a4824-14ed-448d-90d8-bdf6ec96642a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:40.304229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.294491 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/b26a4824-14ed-448d-90d8-bdf6ec96642a-kube-api-access-7src8\") pod \"payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:40.405447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.405403 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:40.424892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.424861 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxc5p\" (UniqueName: \"kubernetes.io/projected/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-kube-api-access-qxc5p\") pod \"payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:40.425336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.425310 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26a4824-14ed-448d-90d8-bdf6ec96642a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:40.425892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.425874 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/b26a4824-14ed-448d-90d8-bdf6ec96642a-kube-api-access-7src8\") pod \"payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:40.426358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.426334 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:40.431250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.428126 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26a4824-14ed-448d-90d8-bdf6ec96642a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:40.437784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.437724 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxc5p\" (UniqueName: \"kubernetes.io/projected/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-kube-api-access-qxc5p\") pod \"payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:40.439108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.439049 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/b26a4824-14ed-448d-90d8-bdf6ec96642a-kube-api-access-7src8\") pod \"payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:40.477411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.475717 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:40.561274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:40.556767 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:41.159001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.158958 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh"] Oct 31 03:38:41.168831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.168588 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j"] Oct 31 03:38:41.689699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.689666 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:38:41.969530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.969428 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" event={"ID":"b26a4824-14ed-448d-90d8-bdf6ec96642a","Type":"ContainerStarted","Data":"66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421"} Oct 31 03:38:41.969530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.969487 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" event={"ID":"b26a4824-14ed-448d-90d8-bdf6ec96642a","Type":"ContainerStarted","Data":"22d9a27e18cad1983dafd67ab2c9ea6135c6875709c4ec6bb7ae489017dd930a"} Oct 31 03:38:41.969530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.969513 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:41.971595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.971567 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" event={"ID":"cf6465fc-e357-4c38-8cd0-c12fb7f45dce","Type":"ContainerStarted","Data":"cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b"} Oct 31 03:38:41.971743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.971612 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" event={"ID":"cf6465fc-e357-4c38-8cd0-c12fb7f45dce","Type":"ContainerStarted","Data":"a2e44a4199a2fd4cdff64132e30b1342f63692d809adacff6661a6ee773a8988"} Oct 31 03:38:41.971743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.971660 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:41.994622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:41.994566 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" podStartSLOduration=2.994552704 podStartE2EDuration="2.994552704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:41.9907443 +0000 UTC m=+1346.185267424" watchObservedRunningTime="2026-10-31 03:38:41.994552704 +0000 UTC m=+1346.189075825" Oct 31 03:38:42.008708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:42.008624 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" podStartSLOduration=3.008607077 podStartE2EDuration="3.008607077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:42.008010205 +0000 UTC m=+1346.202533327" watchObservedRunningTime="2026-10-31 03:38:42.008607077 +0000 UTC m=+1346.203130200" Oct 31 03:38:42.698607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:42.698578 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:38:42.699148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:42.698644 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:38:45.848181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:45.848148 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:38:46.874600 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:46.872398 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:38:47.006004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:46.998364 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj"] Oct 31 03:38:47.014806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.014747 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj"] Oct 31 03:38:47.015137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.015119 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.066054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.064701 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-maas-api-tls\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.066054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.064814 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de472ad7-07c7-495d-8bc6-83ad091f4413-openshift-service-ca\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.066054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.065178 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnz6g\" (UniqueName: \"kubernetes.io/projected/de472ad7-07c7-495d-8bc6-83ad091f4413-kube-api-access-lnz6g\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.066054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.065401 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-metrics-certs\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.067426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.067398 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8361df-serving-cert\"" Oct 31 03:38:47.166862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.166776 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnz6g\" (UniqueName: \"kubernetes.io/projected/de472ad7-07c7-495d-8bc6-83ad091f4413-kube-api-access-lnz6g\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.167035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.166900 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-metrics-certs\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.167096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.167033 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-maas-api-tls\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.167096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.167071 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de472ad7-07c7-495d-8bc6-83ad091f4413-openshift-service-ca\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.167962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.167934 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de472ad7-07c7-495d-8bc6-83ad091f4413-openshift-service-ca\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.170239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.170214 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-metrics-certs\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.171732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.171704 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-maas-api-tls\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.175040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.175014 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnz6g\" (UniqueName: \"kubernetes.io/projected/de472ad7-07c7-495d-8bc6-83ad091f4413-kube-api-access-lnz6g\") pod \"maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:47.371638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:47.371591 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:48.820861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:48.817377 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:48.878229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:48.840441 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:38:48.889350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:48.848252 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 31 03:38:48.948510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:48.945565 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:38:49.005219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.000233 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj"] Oct 31 03:38:49.044255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.038473 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:49.077153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.068240 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:38:49.077153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.074823 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:49.131227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.127576 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:49.251129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.250175 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:49.259646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.259586 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:49.529282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.519482 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:49.544581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.543766 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" event={"ID":"de472ad7-07c7-495d-8bc6-83ad091f4413","Type":"ContainerStarted","Data":"c3181b3c0972562f5dafebc85a6f77cd1745469ff2d9f4a61464c3577e10c056"} Oct 31 03:38:49.892287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.892242 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:49.954167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:49.946129 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:50.079689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.024276 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:50.139828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.048479 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:38:50.139828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.060544 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:50.139828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.081335 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:50.139828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.081390 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:38:50.139828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.137525 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:38:50.139828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.137958 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:38:50.273225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.265238 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:50.280315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.276476 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:38:50.286321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.281854 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:38:50.286321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.282369 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:38:50.555253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.548960 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:38:50.804068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.803294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:38:50.985502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:50.984899 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:51.064762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.064732 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:38:51.079649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.079614 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:38:51.311277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.308681 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:51.311277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.310094 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:38:51.311277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.310469 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:38:51.377554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.375817 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:38:51.494239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.475041 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:38:51.494239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.476422 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:38:51.499439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.499395 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:38:51.502765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.502734 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:38:51.763592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.760528 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" event={"ID":"de472ad7-07c7-495d-8bc6-83ad091f4413","Type":"ContainerStarted","Data":"b3f81a945b339424aacd5173255b227c9e583c4f705eb5a81848d6a4917399bc"} Oct 31 03:38:51.763592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.762175 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:51.850895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.847450 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:51.850895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:51.849365 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:38:52.644540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:52.644503 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:52.675621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:52.675549 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" podStartSLOduration=5.75794609 podStartE2EDuration="6.675512386s" totalImagesPullingTime="917.566296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:52.021258455 +0000 UTC m=+1356.215781578" watchObservedRunningTime="2026-10-31 03:38:52.675512386 +0000 UTC m=+1356.870035508" Oct 31 03:38:52.854802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:52.854752 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:38:52.895092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:52.894989 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:53.027268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.021648 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:53.210287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.178359 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:53.210287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.200304 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:53.223272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.213063 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:38:53.223272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.213186 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc"] Oct 31 03:38:53.226698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.226664 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:38:53.280245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.273302 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:38:53.280245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.273352 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc"] Oct 31 03:38:53.316934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.311352 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:53.330227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.319965 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf"] Oct 31 03:38:53.334656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.334626 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-007d82\"" Oct 31 03:38:53.350402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.350374 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-007d82-dockercfg-66p8j\"" Oct 31 03:38:53.395229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.375555 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10446300-c01e-4947-bcc2-cb29c9a36345-plugins-config-volume\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:53.416866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.416831 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmgx\" (UniqueName: \"kubernetes.io/projected/10446300-c01e-4947-bcc2-cb29c9a36345-kube-api-access-7hmgx\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:53.546642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.535526 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf"] Oct 31 03:38:53.585235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.554681 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:38:53.585235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.566885 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10446300-c01e-4947-bcc2-cb29c9a36345-plugins-config-volume\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:53.585235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.571526 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmgx\" (UniqueName: \"kubernetes.io/projected/10446300-c01e-4947-bcc2-cb29c9a36345-kube-api-access-7hmgx\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:53.585235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.570344 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:53.585235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.567838 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10446300-c01e-4947-bcc2-cb29c9a36345-plugins-config-volume\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:53.588029 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.587994 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:38:53.607630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.597362 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:53.607630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.597668 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:38:53.607630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.597935 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:38:53.684227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.683478 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e0ce96-2b1f-44b2-a657-f7eed6e62911-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:53.684227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.683719 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpssz\" (UniqueName: \"kubernetes.io/projected/60e0ce96-2b1f-44b2-a657-f7eed6e62911-kube-api-access-kpssz\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:53.788224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.787187 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e0ce96-2b1f-44b2-a657-f7eed6e62911-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:53.792224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.788508 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e0ce96-2b1f-44b2-a657-f7eed6e62911-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:53.817281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.802882 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpssz\" (UniqueName: \"kubernetes.io/projected/60e0ce96-2b1f-44b2-a657-f7eed6e62911-kube-api-access-kpssz\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:53.834142 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.834095 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpssz\" (UniqueName: \"kubernetes.io/projected/60e0ce96-2b1f-44b2-a657-f7eed6e62911-kube-api-access-kpssz\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:53.919229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.908290 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:53.946248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.946185 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:53.957236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.949468 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:38:53.964477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.964447 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmgx\" (UniqueName: \"kubernetes.io/projected/10446300-c01e-4947-bcc2-cb29c9a36345-kube-api-access-7hmgx\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:53.970099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:53.970075 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:38:54.060525 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.060495 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:54.139890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.139795 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:54.276394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.276343 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs"] Oct 31 03:38:54.296558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.296529 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.326150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.326031 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:38:54.353317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.353281 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.353622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.353600 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj96\" (UniqueName: \"kubernetes.io/projected/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-kube-api-access-kdj96\") pod \"payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.405484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.392288 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a3b986-dockercfg-5r562\"" Oct 31 03:38:54.448513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.417831 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a3b986\"" Oct 31 03:38:54.489346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.462519 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.489346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.462613 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj96\" (UniqueName: \"kubernetes.io/projected/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-kube-api-access-kdj96\") pod \"payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.489346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.464370 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.647390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.643299 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs"] Oct 31 03:38:54.698393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.697868 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj96\" (UniqueName: \"kubernetes.io/projected/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-kube-api-access-kdj96\") pod \"payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.704290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.700613 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:54.991312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:54.990080 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:55.112838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.112796 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:38:55.165626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.140725 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:38:55.166828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.166682 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr"] Oct 31 03:38:55.216601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.197921 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:38:55.302759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.274758 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:55.403112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.398354 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:55.414341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.413036 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:55.414341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.413878 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:38:55.429228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.426240 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:55.553663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.553618 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:55.644428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.606902 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:55.681225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.649584 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr"] Oct 31 03:38:55.681225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:55.657621 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:56.087255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.084639 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjl4\" (UniqueName: \"kubernetes.io/projected/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-kube-api-access-5tjl4\") pod \"payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:56.087255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.085072 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:56.101730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.101580 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:56.218312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.148311 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:56.324574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.321312 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:38:56.339513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.339481 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:38:56.347547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.347513 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:38:56.350455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.350427 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:38:56.361152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.361104 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:56.361823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.361773 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjl4\" (UniqueName: \"kubernetes.io/projected/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-kube-api-access-5tjl4\") pod \"payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:56.376696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.374726 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:56.380421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.377780 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:38:56.417934 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:38:56.417896 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10446300_c01e_4947_bcc2_cb29c9a36345.slice/crio-0132ebf31da7b8f598c5b60c554bdff022b12b560772513374f1c199aafbe164 WatchSource:0}: Error finding container 0132ebf31da7b8f598c5b60c554bdff022b12b560772513374f1c199aafbe164: Status 404 returned error can't find the container with id 0132ebf31da7b8f598c5b60c554bdff022b12b560772513374f1c199aafbe164 Oct 31 03:38:56.428234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.427387 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:38:56.430884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.429601 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:38:56.430884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.430066 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:38:56.430884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.430149 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:38:56.448231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.444447 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:56.455932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.451016 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:38:56.494503 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.459167 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf"] Oct 31 03:38:56.525268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.496957 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:38:56.650313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.643639 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:38:56.650313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.644602 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:38:56.658777 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.658751 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjl4\" (UniqueName: \"kubernetes.io/projected/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-kube-api-access-5tjl4\") pod \"payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:56.754994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:56.753713 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:57.047304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.047084 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:38:57.138233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.134494 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:57.172982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.172942 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:38:57.173599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.173256 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc"] Oct 31 03:38:57.174650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.174633 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:38:57.263258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.229683 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:38:57.263258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.251176 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:38:57.308310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.299235 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:38:57.336079 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.336044 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:38:57.428351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.425563 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" event={"ID":"60e0ce96-2b1f-44b2-a657-f7eed6e62911","Type":"ContainerStarted","Data":"e043eb8cca4a40027877540fae5138873784a43034fc4650d4c66c66f3ff6127"} Oct 31 03:38:57.432127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.432088 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" event={"ID":"10446300-c01e-4947-bcc2-cb29c9a36345","Type":"ContainerStarted","Data":"0132ebf31da7b8f598c5b60c554bdff022b12b560772513374f1c199aafbe164"} Oct 31 03:38:57.434878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.434843 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" event={"ID":"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98","Type":"ContainerStarted","Data":"f29b1001112b5ae1ae90c2a735e782e4f2eef3ccf58a7b74dad667e33b71bb3e"} Oct 31 03:38:57.438430 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:57.438406 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs"] Oct 31 03:38:58.207581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.207534 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:38:58.353834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.353783 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr"] Oct 31 03:38:58.511231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.507474 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" event={"ID":"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49","Type":"ContainerStarted","Data":"4f845f47b1f83ee34fc3c969118ec04a2697bab9b0d86c82ae3e66792da99f2a"} Oct 31 03:38:58.511231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.510785 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" event={"ID":"60e0ce96-2b1f-44b2-a657-f7eed6e62911","Type":"ContainerStarted","Data":"1114c161b3dffab14ca3c055c67b52a60f90909882b5d9aec2e7cc24fb4b629c"} Oct 31 03:38:58.517229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.511998 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:38:58.517229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.514408 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" event={"ID":"10446300-c01e-4947-bcc2-cb29c9a36345","Type":"ContainerStarted","Data":"95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce"} Oct 31 03:38:58.517229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.515269 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:38:58.522323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.519717 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" event={"ID":"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98","Type":"ContainerStarted","Data":"5aecb8a139708fc985f8caae54a3e7a3a53f986086f8882e15a20881e336568b"} Oct 31 03:38:58.522323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.520551 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:38:58.559608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.552169 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" podStartSLOduration=6.552150531 podStartE2EDuration="6.552150531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:58.544489893 +0000 UTC m=+1362.739013028" watchObservedRunningTime="2026-10-31 03:38:58.552150531 +0000 UTC m=+1362.746673659" Oct 31 03:38:58.605672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.601466 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" podStartSLOduration=4.601445979 podStartE2EDuration="4.601445979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:58.594011774 +0000 UTC m=+1362.788534895" watchObservedRunningTime="2026-10-31 03:38:58.601445979 +0000 UTC m=+1362.795969101" Oct 31 03:38:58.650487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:58.650428 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" podStartSLOduration=6.650409316 podStartE2EDuration="6.650409316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:58.646064773 +0000 UTC m=+1362.840587919" watchObservedRunningTime="2026-10-31 03:38:58.650409316 +0000 UTC m=+1362.844932437" Oct 31 03:38:59.530523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:59.530469 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" event={"ID":"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49","Type":"ContainerStarted","Data":"912f4fd67801256dfccae4bdcc9c4ad6e4055e05c69e3f0bbae94afd984fce00"} Oct 31 03:38:59.531147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:59.530930 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:38:59.557894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:38:59.557829 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" podStartSLOduration=5.557810701 podStartE2EDuration="5.557810701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:59.550621623 +0000 UTC m=+1363.745144753" watchObservedRunningTime="2026-10-31 03:38:59.557810701 +0000 UTC m=+1363.752333830" Oct 31 03:39:01.421099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:01.421057 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:39:01.421620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:01.421112 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:03.694492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.691492 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482"] Oct 31 03:39:03.696875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.696854 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.726167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.724730 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482"] Oct 31 03:39:03.817746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.817703 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.817956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.817784 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.818058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.818027 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2q4c\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-kube-api-access-d2q4c\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.818308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.818280 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.818444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.818419 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e461da26-40b4-4633-b9fc-bb23d74c9449-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.818551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.818494 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.818551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.818534 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.818712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.818674 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.818784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.818717 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920169 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920274 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920306 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920507 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920596 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920689 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920717 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2q4c\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-kube-api-access-d2q4c\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920850 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920833 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920857 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.920914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920893 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e461da26-40b4-4633-b9fc-bb23d74c9449-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.921023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920920 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.921023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920928 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.921023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.920832 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.921650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.921626 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e461da26-40b4-4633-b9fc-bb23d74c9449-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.923543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.923511 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.923816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.923793 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.932164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.932129 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:03.932432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:03.932406 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2q4c\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-kube-api-access-d2q4c\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-q7482\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:04.033397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.033324 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:04.446531 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:39:04.446491 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode461da26_40b4_4633_b9fc_bb23d74c9449.slice/crio-44a092fd35ebf11a773225c60d40458991018371b68405cb8e89f9b2b366852f WatchSource:0}: Error finding container 44a092fd35ebf11a773225c60d40458991018371b68405cb8e89f9b2b366852f: Status 404 returned error can't find the container with id 44a092fd35ebf11a773225c60d40458991018371b68405cb8e89f9b2b366852f Oct 31 03:39:04.446811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.446788 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482"] Oct 31 03:39:04.450097 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.450058 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:04.450214 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.450146 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:04.450214 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.450186 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:04.643269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.643236 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" event={"ID":"e461da26-40b4-4633-b9fc-bb23d74c9449","Type":"ContainerStarted","Data":"44a092fd35ebf11a773225c60d40458991018371b68405cb8e89f9b2b366852f"} Oct 31 03:39:04.645354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.645328 5520 generic.go:356] "Generic (PLEG): container finished" podID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerID="6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253" exitCode=0 Oct 31 03:39:04.645487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.645375 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" event={"ID":"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3","Type":"ContainerDied","Data":"6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253"} Oct 31 03:39:04.645843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.645828 5520 scope.go:122] "RemoveContainer" containerID="6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253" Oct 31 03:39:04.647869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.647839 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:04.647973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.647931 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:04.648041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:04.647980 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:05.854245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:05.838927 5520 generic.go:356] "Generic (PLEG): container finished" podID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerID="482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90" exitCode=0 Oct 31 03:39:05.854245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:05.839193 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" event={"ID":"44a96e65-fb7f-4bc9-88f7-2bf66a97d460","Type":"ContainerDied","Data":"482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90"} Oct 31 03:39:05.854245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:05.839927 5520 scope.go:122] "RemoveContainer" containerID="482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90" Oct 31 03:39:05.874070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:05.874010 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:05.874471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:05.874438 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:05.874775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:05.874686 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:05.881991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:05.881963 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" event={"ID":"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3","Type":"ContainerStarted","Data":"692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af"} Oct 31 03:39:06.220009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.215547 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:06.220009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.215681 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:06.220009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.219314 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:06.220009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.219362 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:06.595521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.565937 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:06.595521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.566080 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:39:06.595521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.566104 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" event={"ID":"e461da26-40b4-4633-b9fc-bb23d74c9449","Type":"ContainerStarted","Data":"81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81"} Oct 31 03:39:06.681231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.673877 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podStartSLOduration=3.673856094 podStartE2EDuration="3.673856094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:06.65505431 +0000 UTC m=+1370.849577431" watchObservedRunningTime="2026-10-31 03:39:06.673856094 +0000 UTC m=+1370.868379216" Oct 31 03:39:06.839229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:06.838373 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:39:07.030225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.025356 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" event={"ID":"44a96e65-fb7f-4bc9-88f7-2bf66a97d460","Type":"ContainerStarted","Data":"a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51"} Oct 31 03:39:07.041243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.040333 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:07.041243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.040391 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:07.129227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.127140 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:07.129227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.127231 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:07.561188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.555690 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:39:07.561188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.557927 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:07.561188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.557992 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:07.916575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.916470 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:07.960345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:07.960303 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:08.050374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.050337 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:08.050880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.050395 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:08.071397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.071352 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:39:08.096223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.095923 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:08.124107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.120669 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:08.124107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.120727 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:08.234349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.234288 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:08.506371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.506273 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:08.555650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.555616 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:08.555927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:08.555904 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:09.048232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:09.039309 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:09.048232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:09.039411 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:09.133653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:09.133620 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:09.134248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:09.134218 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:09.493821 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:09.493782 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:39:09.565001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:09.564686 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:09.565001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:09.564745 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:10.031477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.031436 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:10.061915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.061879 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:10.062192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.062164 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:10.173228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.140157 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:10.173228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.140253 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:10.257723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.257683 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:10.389284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.385785 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:10.548158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.548116 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:10.560417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.560359 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:10.565511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.565485 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:39:10.589396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.589360 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:39:10.591413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.591388 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:10.591634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.591611 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:10.593636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.593617 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:39:10.605485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.605458 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:39:11.072275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:10.993628 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:11.241493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.240303 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:11.270224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.256526 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:11.270224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.256585 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:11.270224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.256993 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:11.270224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.257030 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:11.352282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.352133 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:39:11.397628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.352525 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:39:11.566303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.563751 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:39:11.566303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.564298 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:39:11.607592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.607495 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:11.607859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.607829 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:11.962667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:11.954515 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.048233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.045883 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.110230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.077500 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.114604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.098932 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:39:12.142714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.098996 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.260625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.258990 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.261747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.261429 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:12.261998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.261959 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:12.266733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.266699 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:12.266933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.266913 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:12.352424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.352380 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.353103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.352733 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.396345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.396300 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.396993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.396947 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:39:12.442226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.422179 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:39:12.442226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.422273 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:39:12.442226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.422310 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:39:12.442226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.422339 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:39:12.502634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.484320 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:12.551402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.551307 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:39:12.553685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.553659 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:39:12.594018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.593053 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:12.594018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:12.593115 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:13.105253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.055309 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:13.105253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.066324 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:39:13.105253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.067704 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:13.105253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.067751 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:13.145266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.119600 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:39:13.145266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.119925 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:39:13.146728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.146305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:39:13.159407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.159367 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:39:13.159997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.159975 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:39:13.199847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.199809 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:13.200122 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.200089 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:13.316185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.316144 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:39:13.357642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.336677 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:13.441716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.357881 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:13.442518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.359185 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:39:13.443272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.401326 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:39:13.444613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.401363 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:39:13.444838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.444818 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:39:13.445023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.444996 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:39:13.490904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.490862 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:13.495229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.491605 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:39:13.954578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.940751 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:13.968371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.968328 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:13.972551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.972512 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:39:14.286496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:13.982395 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:14.287476 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.149577 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:14.288691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.215288 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:39:14.289485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.240184 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:14.289701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.289657 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:14.289843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.272503 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:14.290296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.242389 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:14.291016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.268092 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:14.291189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.291166 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:14.353391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.346181 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:39:14.380256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.380195 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:14.710277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.451635 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:39:15.186062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.451703 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:15.186975 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.451741 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:39:15.223332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.451776 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:39:15.231273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.538162 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:15.231475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:14.766543 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:15.246448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.246411 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:15.247089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.117601 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:15.247581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.089276 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 31 03:39:15.247749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.247727 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 31 03:39:15.247857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.130852 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:39:15.247965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.247947 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 31 03:39:15.338063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.338026 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 31 03:39:15.338453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.338411 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:39:15.338949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.338923 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:39:15.339427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.339402 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:39:15.339957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.339936 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:39:15.340524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.340502 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:39:15.341048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.341027 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:39:15.386228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.365438 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:15.400398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.400348 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 31 03:39:15.419706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.419664 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:15.565142 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.565049 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:15.565453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.565426 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:15.606407 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:15.606368 5520 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.666s" Oct 31 03:39:15.606680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.606666 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:39:15.606936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.606922 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:39:15.607093 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607071 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:39:15.607237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607226 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:39:15.607351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607341 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:39:15.607482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607472 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:39:15.607589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607580 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:39:15.607690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607681 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:39:15.607788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607779 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:39:15.607887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.607878 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:39:15.661272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.656646 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:39:15.749918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.749889 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:39:15.783462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.783431 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:39:15.847385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.803382 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:39:15.943003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.942975 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:39:15.963746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:15.963458 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:39:16.141431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.141329 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:39:16.384231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.377332 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:16.420634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.391792 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:16.444341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.444304 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:39:16.561415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.561386 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:16.562724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.562654 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:16.639317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.639184 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:39:16.648571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:16.648544 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:39:17.554848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:17.554808 5520 patch_prober.go:29] interesting pod/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:39:17.555379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:17.554891 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 31 03:39:18.559569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.559537 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:39:18.560748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.560725 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:39:18.582590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.582558 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9"] Oct 31 03:39:18.590761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.590728 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.601705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.601677 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:39:18.602048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.602022 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:39:18.614730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.614704 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9"] Oct 31 03:39:18.721096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.721060 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.721096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.721096 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.721513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.721493 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hkb\" (UniqueName: \"kubernetes.io/projected/085a7ddc-b35e-41c7-86cf-b594d25c7703-kube-api-access-64hkb\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.824057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.823939 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hkb\" (UniqueName: \"kubernetes.io/projected/085a7ddc-b35e-41c7-86cf-b594d25c7703-kube-api-access-64hkb\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.824518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.824497 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.824724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.824705 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.826401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.826364 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.892223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.886263 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hkb\" (UniqueName: \"kubernetes.io/projected/085a7ddc-b35e-41c7-86cf-b594d25c7703-kube-api-access-64hkb\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.892825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.892796 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:18.923313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:18.922106 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:19.426255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:19.419582 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9"] Oct 31 03:39:20.324839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:20.324801 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" event={"ID":"085a7ddc-b35e-41c7-86cf-b594d25c7703","Type":"ContainerStarted","Data":"61932498769f2274086f5f14d46ca8e833228714c44c20fc0e8ecd93b7da8a2c"} Oct 31 03:39:20.325272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:20.324857 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" event={"ID":"085a7ddc-b35e-41c7-86cf-b594d25c7703","Type":"ContainerStarted","Data":"984d4fef319fd9f6b32f97d287eb44051c1b5b4b301c7353ddb840047160b89d"} Oct 31 03:39:20.344697 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:20.344644 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" podStartSLOduration=2.344630307 podStartE2EDuration="2.344630307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:20.340272943 +0000 UTC m=+1384.534796071" watchObservedRunningTime="2026-10-31 03:39:20.344630307 +0000 UTC m=+1384.539153431" Oct 31 03:39:21.335757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:21.335723 5520 generic.go:356] "Generic (PLEG): container finished" podID="085a7ddc-b35e-41c7-86cf-b594d25c7703" containerID="61932498769f2274086f5f14d46ca8e833228714c44c20fc0e8ecd93b7da8a2c" exitCode=0 Oct 31 03:39:21.336158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:21.335792 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" event={"ID":"085a7ddc-b35e-41c7-86cf-b594d25c7703","Type":"ContainerDied","Data":"61932498769f2274086f5f14d46ca8e833228714c44c20fc0e8ecd93b7da8a2c"} Oct 31 03:39:22.554096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.554069 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:22.693901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.693867 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-xks-ca\") pod \"085a7ddc-b35e-41c7-86cf-b594d25c7703\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " Oct 31 03:39:22.694077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.694031 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085a7ddc-b35e-41c7-86cf-b594d25c7703-kube-api-access-64hkb\" (UniqueName: \"kubernetes.io/projected/085a7ddc-b35e-41c7-86cf-b594d25c7703-kube-api-access-64hkb\") pod \"085a7ddc-b35e-41c7-86cf-b594d25c7703\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " Oct 31 03:39:22.694129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.694100 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-openshift-ca\") pod \"085a7ddc-b35e-41c7-86cf-b594d25c7703\" (UID: \"085a7ddc-b35e-41c7-86cf-b594d25c7703\") " Oct 31 03:39:22.694595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.694564 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-openshift-ca" pod "085a7ddc-b35e-41c7-86cf-b594d25c7703" (UID: "085a7ddc-b35e-41c7-86cf-b594d25c7703"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:22.695231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.695188 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:22.696489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.696463 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-xks-ca" pod "085a7ddc-b35e-41c7-86cf-b594d25c7703" (UID: "085a7ddc-b35e-41c7-86cf-b594d25c7703"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:22.696934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.696896 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085a7ddc-b35e-41c7-86cf-b594d25c7703-kube-api-access-64hkb" pod "085a7ddc-b35e-41c7-86cf-b594d25c7703" (UID: "085a7ddc-b35e-41c7-86cf-b594d25c7703"). InnerVolumeSpecName "kube-api-access-64hkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:22.800641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.800596 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085a7ddc-b35e-41c7-86cf-b594d25c7703-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:22.800641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:22.800637 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hkb\" (UniqueName: \"kubernetes.io/projected/085a7ddc-b35e-41c7-86cf-b594d25c7703-kube-api-access-64hkb\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:23.356021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:23.355977 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" Oct 31 03:39:23.358859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:23.356116 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9" event={"ID":"085a7ddc-b35e-41c7-86cf-b594d25c7703","Type":"ContainerDied","Data":"984d4fef319fd9f6b32f97d287eb44051c1b5b4b301c7353ddb840047160b89d"} Oct 31 03:39:23.358859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:23.356175 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="984d4fef319fd9f6b32f97d287eb44051c1b5b4b301c7353ddb840047160b89d" Oct 31 03:39:25.324227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:25.317305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:28.713714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:28.713673 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq"] Oct 31 03:39:28.736185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:28.736153 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj"] Oct 31 03:39:29.053669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:29.053585 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc"] Oct 31 03:39:29.345275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:29.335613 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb"] Oct 31 03:39:33.719255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:33.719092 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" podUID="50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" containerName="payload-processing" containerID="cri-o://8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5" gracePeriod=25 Oct 31 03:39:33.736717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:33.736682 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" podUID="fdb96d9d-566b-4d00-9c59-857a2d53552f" containerName="payload-pre-processing" containerID="cri-o://efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203" gracePeriod=25 Oct 31 03:39:34.054867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.054246 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" podUID="891e5f8d-1360-406b-894d-8cb637655611" containerName="payload-processing" containerID="cri-o://beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d" gracePeriod=25 Oct 31 03:39:34.077792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.067565 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:39:34.092576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.092541 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-plugins-config-volume\") pod \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " Oct 31 03:39:34.092829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.092809 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-kube-api-access-tbdfh\" (UniqueName: \"kubernetes.io/projected/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-kube-api-access-tbdfh\") pod \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\" (UID: \"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7\") " Oct 31 03:39:34.093044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.093016 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-plugins-config-volume" pod "50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" (UID: "50bb7aaf-5529-4cd5-ace6-1f8e89b243c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:34.093705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.093678 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.095145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.095121 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-kube-api-access-tbdfh" pod "50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" (UID: "50bb7aaf-5529-4cd5-ace6-1f8e89b243c7"). InnerVolumeSpecName "kube-api-access-tbdfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:34.099518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.099492 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6"] Oct 31 03:39:34.100586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.100554 5520 cpu_manager.go:414] "containerMap: removing container" podUID="50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" containerName="payload-processing" Oct 31 03:39:34.100586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.100576 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" containerName="payload-processing" Oct 31 03:39:34.101015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.100613 5520 cpu_manager.go:414] "containerMap: removing container" podUID="085a7ddc-b35e-41c7-86cf-b594d25c7703" containerName="revoke-keys" Oct 31 03:39:34.101015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.100621 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085a7ddc-b35e-41c7-86cf-b594d25c7703" containerName="revoke-keys" Oct 31 03:39:34.101015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.100988 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="085a7ddc-b35e-41c7-86cf-b594d25c7703" containerName="revoke-keys" Oct 31 03:39:34.101175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.101021 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" containerName="payload-processing" Oct 31 03:39:34.105642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.105385 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.112159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.112013 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c5dcb7\"" Oct 31 03:39:34.115264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.114460 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6"] Oct 31 03:39:34.173863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.173516 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:39:34.212497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.207085 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c5dcb7-dockercfg-ddk9w\"" Oct 31 03:39:34.246240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.243964 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb96d9d-566b-4d00-9c59-857a2d53552f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb96d9d-566b-4d00-9c59-857a2d53552f-plugins-config-volume\") pod \"fdb96d9d-566b-4d00-9c59-857a2d53552f\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " Oct 31 03:39:34.246240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.244255 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb96d9d-566b-4d00-9c59-857a2d53552f-kube-api-access-t4x5q\" (UniqueName: \"kubernetes.io/projected/fdb96d9d-566b-4d00-9c59-857a2d53552f-kube-api-access-t4x5q\") pod \"fdb96d9d-566b-4d00-9c59-857a2d53552f\" (UID: \"fdb96d9d-566b-4d00-9c59-857a2d53552f\") " Oct 31 03:39:34.246240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.244773 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpn56\" (UniqueName: \"kubernetes.io/projected/23090f24-1e03-4f47-8fc6-3ccdaae08883-kube-api-access-mpn56\") pod \"payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.246240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.244895 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23090f24-1e03-4f47-8fc6-3ccdaae08883-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.246240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.245429 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbdfh\" (UniqueName: \"kubernetes.io/projected/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7-kube-api-access-tbdfh\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.277279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.275095 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll"] Oct 31 03:39:34.277279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.276724 5520 cpu_manager.go:414] "containerMap: removing container" podUID="fdb96d9d-566b-4d00-9c59-857a2d53552f" containerName="payload-pre-processing" Oct 31 03:39:34.277279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.276749 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb96d9d-566b-4d00-9c59-857a2d53552f" containerName="payload-pre-processing" Oct 31 03:39:34.277279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.276932 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb96d9d-566b-4d00-9c59-857a2d53552f" containerName="payload-pre-processing" Oct 31 03:39:34.282227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.279320 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb96d9d-566b-4d00-9c59-857a2d53552f-plugins-config-volume" pod "fdb96d9d-566b-4d00-9c59-857a2d53552f" (UID: "fdb96d9d-566b-4d00-9c59-857a2d53552f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:34.282227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.281858 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.347267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.336367 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" podUID="c6143c03-8a03-4e2b-ac02-18ac593c103f" containerName="payload-pre-processing" containerID="cri-o://07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d" gracePeriod=25 Oct 31 03:39:34.347267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.337257 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb96d9d-566b-4d00-9c59-857a2d53552f-kube-api-access-t4x5q" pod "fdb96d9d-566b-4d00-9c59-857a2d53552f" (UID: "fdb96d9d-566b-4d00-9c59-857a2d53552f"). InnerVolumeSpecName "kube-api-access-t4x5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:34.350048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.348528 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll"] Oct 31 03:39:34.355260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.351401 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpn56\" (UniqueName: \"kubernetes.io/projected/23090f24-1e03-4f47-8fc6-3ccdaae08883-kube-api-access-mpn56\") pod \"payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.355260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.351517 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23090f24-1e03-4f47-8fc6-3ccdaae08883-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.355260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.351802 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4x5q\" (UniqueName: \"kubernetes.io/projected/fdb96d9d-566b-4d00-9c59-857a2d53552f-kube-api-access-t4x5q\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.355260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.351822 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb96d9d-566b-4d00-9c59-857a2d53552f-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.355260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.352679 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23090f24-1e03-4f47-8fc6-3ccdaae08883-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.363897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.363869 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpn56\" (UniqueName: \"kubernetes.io/projected/23090f24-1e03-4f47-8fc6-3ccdaae08883-kube-api-access-mpn56\") pod \"payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.456629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.456589 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffm6\" (UniqueName: \"kubernetes.io/projected/5a993c9e-f98b-4d94-81ee-6c51e6eda148-kube-api-access-fffm6\") pod \"payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.456811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.456680 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a993c9e-f98b-4d94-81ee-6c51e6eda148-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.462116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.461801 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:34.494672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.494284 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:39:34.561060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.561025 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a993c9e-f98b-4d94-81ee-6c51e6eda148-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.561538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.561511 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffm6\" (UniqueName: \"kubernetes.io/projected/5a993c9e-f98b-4d94-81ee-6c51e6eda148-kube-api-access-fffm6\") pod \"payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.563272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.562754 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a993c9e-f98b-4d94-81ee-6c51e6eda148-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.572609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.571888 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffm6\" (UniqueName: \"kubernetes.io/projected/5a993c9e-f98b-4d94-81ee-6c51e6eda148-kube-api-access-fffm6\") pod \"payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.614225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.613633 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:34.681261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.666878 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891e5f8d-1360-406b-894d-8cb637655611-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891e5f8d-1360-406b-894d-8cb637655611-plugins-config-volume\") pod \"891e5f8d-1360-406b-894d-8cb637655611\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " Oct 31 03:39:34.681261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.667102 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891e5f8d-1360-406b-894d-8cb637655611-kube-api-access-vqfqs\" (UniqueName: \"kubernetes.io/projected/891e5f8d-1360-406b-894d-8cb637655611-kube-api-access-vqfqs\") pod \"891e5f8d-1360-406b-894d-8cb637655611\" (UID: \"891e5f8d-1360-406b-894d-8cb637655611\") " Oct 31 03:39:34.681261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.668624 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891e5f8d-1360-406b-894d-8cb637655611-plugins-config-volume" pod "891e5f8d-1360-406b-894d-8cb637655611" (UID: "891e5f8d-1360-406b-894d-8cb637655611"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:34.681261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.671019 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891e5f8d-1360-406b-894d-8cb637655611-kube-api-access-vqfqs" pod "891e5f8d-1360-406b-894d-8cb637655611" (UID: "891e5f8d-1360-406b-894d-8cb637655611"). InnerVolumeSpecName "kube-api-access-vqfqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:34.723249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.723163 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:39:34.724064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.724040 5520 generic.go:356] "Generic (PLEG): container finished" podID="c6143c03-8a03-4e2b-ac02-18ac593c103f" containerID="07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d" exitCode=0 Oct 31 03:39:34.724189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.724161 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" event={"ID":"c6143c03-8a03-4e2b-ac02-18ac593c103f","Type":"ContainerDied","Data":"07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d"} Oct 31 03:39:34.724282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.724246 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" event={"ID":"c6143c03-8a03-4e2b-ac02-18ac593c103f","Type":"ContainerDied","Data":"c62c57ea35b7c13f545e970970957529cd3d507f6ac7a3d52d160223c2763521"} Oct 31 03:39:34.724342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.724281 5520 scope.go:122] "RemoveContainer" containerID="07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d" Oct 31 03:39:34.728782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.726739 5520 generic.go:356] "Generic (PLEG): container finished" podID="891e5f8d-1360-406b-894d-8cb637655611" containerID="beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d" exitCode=0 Oct 31 03:39:34.728782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.726840 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" Oct 31 03:39:34.728782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.728060 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" event={"ID":"891e5f8d-1360-406b-894d-8cb637655611","Type":"ContainerDied","Data":"beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d"} Oct 31 03:39:34.728782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.728099 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc" event={"ID":"891e5f8d-1360-406b-894d-8cb637655611","Type":"ContainerDied","Data":"ffc21c493892f1d7a5f868c9bd806f44e8ebc87e9fcaf24239e5d451b411537f"} Oct 31 03:39:34.732033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.732004 5520 generic.go:356] "Generic (PLEG): container finished" podID="fdb96d9d-566b-4d00-9c59-857a2d53552f" containerID="efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203" exitCode=0 Oct 31 03:39:34.732175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.732148 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" event={"ID":"fdb96d9d-566b-4d00-9c59-857a2d53552f","Type":"ContainerDied","Data":"efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203"} Oct 31 03:39:34.732273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.732217 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" event={"ID":"fdb96d9d-566b-4d00-9c59-857a2d53552f","Type":"ContainerDied","Data":"15d828ba6a4304b2f5be130293adfbf8c89d6b5ca1e0d184af160ab5f9cc2a6c"} Oct 31 03:39:34.732340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.732319 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj" Oct 31 03:39:34.740664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.735270 5520 generic.go:356] "Generic (PLEG): container finished" podID="50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" containerID="8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5" exitCode=0 Oct 31 03:39:34.740664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.735305 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" event={"ID":"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7","Type":"ContainerDied","Data":"8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5"} Oct 31 03:39:34.740664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.735342 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" event={"ID":"50bb7aaf-5529-4cd5-ace6-1f8e89b243c7","Type":"ContainerDied","Data":"5fd7465a066b5d4c7306d6cd8316f09ab8957afa41a6a8207f160b95fadce318"} Oct 31 03:39:34.740664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.735422 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq" Oct 31 03:39:34.782610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.769681 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqfqs\" (UniqueName: \"kubernetes.io/projected/891e5f8d-1360-406b-894d-8cb637655611-kube-api-access-vqfqs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.782610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.769708 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891e5f8d-1360-406b-894d-8cb637655611-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.787045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.787010 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6"] Oct 31 03:39:34.866309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.863505 5520 scope.go:122] "RemoveContainer" containerID="07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d" Oct 31 03:39:34.874031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.873998 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6143c03-8a03-4e2b-ac02-18ac593c103f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6143c03-8a03-4e2b-ac02-18ac593c103f-plugins-config-volume\") pod \"c6143c03-8a03-4e2b-ac02-18ac593c103f\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " Oct 31 03:39:34.874183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.874075 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6143c03-8a03-4e2b-ac02-18ac593c103f-kube-api-access-qz7j6\" (UniqueName: \"kubernetes.io/projected/c6143c03-8a03-4e2b-ac02-18ac593c103f-kube-api-access-qz7j6\") pod \"c6143c03-8a03-4e2b-ac02-18ac593c103f\" (UID: \"c6143c03-8a03-4e2b-ac02-18ac593c103f\") " Oct 31 03:39:34.876572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.876541 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6143c03-8a03-4e2b-ac02-18ac593c103f-plugins-config-volume" pod "c6143c03-8a03-4e2b-ac02-18ac593c103f" (UID: "c6143c03-8a03-4e2b-ac02-18ac593c103f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:34.911235 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:34.890332 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d\": container with ID starting with 07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d not found: ID does not exist" containerID="07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d" Oct 31 03:39:34.911235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.890378 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d"} err="failed to get container status \"07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d\": rpc error: code = NotFound desc = could not find container \"07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d\": container with ID starting with 07bc34112871112661582c4ad1c199bf9aa5eec368dff9026d8b168ccacaaf0d not found: ID does not exist" Oct 31 03:39:34.911235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.890404 5520 scope.go:122] "RemoveContainer" containerID="beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d" Oct 31 03:39:34.911235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.895861 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6143c03-8a03-4e2b-ac02-18ac593c103f-kube-api-access-qz7j6" pod "c6143c03-8a03-4e2b-ac02-18ac593c103f" (UID: "c6143c03-8a03-4e2b-ac02-18ac593c103f"). InnerVolumeSpecName "kube-api-access-qz7j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:34.928575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.927812 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq"] Oct 31 03:39:34.934828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.934752 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-555bd6-7bc4f86d4f-fjndq"] Oct 31 03:39:34.941702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.941646 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc"] Oct 31 03:39:34.960011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.959981 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-adb7b7-65cd69d77c-rkbfc"] Oct 31 03:39:34.966279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.966256 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj"] Oct 31 03:39:34.971240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.971138 5520 scope.go:122] "RemoveContainer" containerID="beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d" Oct 31 03:39:34.971794 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:34.971661 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d\": container with ID starting with beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d not found: ID does not exist" containerID="beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d" Oct 31 03:39:34.971794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.971694 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d"} err="failed to get container status \"beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d\": rpc error: code = NotFound desc = could not find container \"beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d\": container with ID starting with beb42ec7fcfc86fa9a36d0a7afe7a388dd322aef756cd7668bea3c22ddbc541d not found: ID does not exist" Oct 31 03:39:34.971794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.971716 5520 scope.go:122] "RemoveContainer" containerID="efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203" Oct 31 03:39:34.974641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.974475 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-555bd6-75bb699c9f-st9nj"] Oct 31 03:39:34.978480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.978459 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6143c03-8a03-4e2b-ac02-18ac593c103f-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.978564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.978486 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz7j6\" (UniqueName: \"kubernetes.io/projected/c6143c03-8a03-4e2b-ac02-18ac593c103f-kube-api-access-qz7j6\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:34.996957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.996885 5520 scope.go:122] "RemoveContainer" containerID="efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203" Oct 31 03:39:34.997577 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:34.997414 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203\": container with ID starting with efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203 not found: ID does not exist" containerID="efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203" Oct 31 03:39:34.997577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.997462 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203"} err="failed to get container status \"efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203\": rpc error: code = NotFound desc = could not find container \"efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203\": container with ID starting with efdf5f335206801f752c4ca7cdb3b56a3e808c04569fcce99ab2882f8d610203 not found: ID does not exist" Oct 31 03:39:34.997577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:34.997489 5520 scope.go:122] "RemoveContainer" containerID="8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5" Oct 31 03:39:35.041822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.041681 5520 scope.go:122] "RemoveContainer" containerID="8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5" Oct 31 03:39:35.042299 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:35.042191 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5\": container with ID starting with 8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5 not found: ID does not exist" containerID="8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5" Oct 31 03:39:35.042299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.042255 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5"} err="failed to get container status \"8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5\": rpc error: code = NotFound desc = could not find container \"8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5\": container with ID starting with 8567fc8eb8de1c98f75da1be5be4067f47519105a003978d1cfcf7ce0f337db5 not found: ID does not exist" Oct 31 03:39:35.178274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.178231 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll"] Oct 31 03:39:35.745688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.745646 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" event={"ID":"23090f24-1e03-4f47-8fc6-3ccdaae08883","Type":"ContainerStarted","Data":"fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd"} Oct 31 03:39:35.746178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.745717 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:35.746178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.745743 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" event={"ID":"23090f24-1e03-4f47-8fc6-3ccdaae08883","Type":"ContainerStarted","Data":"a46c91f487b373ee382271a9569bf13219cfb940a360fa4e650d1e45ca0677d9"} Oct 31 03:39:35.746995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.746976 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb" Oct 31 03:39:35.752115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.752008 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" event={"ID":"5a993c9e-f98b-4d94-81ee-6c51e6eda148","Type":"ContainerStarted","Data":"2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3"} Oct 31 03:39:35.752439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.752063 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" event={"ID":"5a993c9e-f98b-4d94-81ee-6c51e6eda148","Type":"ContainerStarted","Data":"623efa4d33dc01f89bc5cfefba4248b6c47e7a8cde16e0cfb593f1ccb0ef6b00"} Oct 31 03:39:35.752439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.752396 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:35.783477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.783416 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" podStartSLOduration=18.78339434 podStartE2EDuration="18.78339434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:35.763773491 +0000 UTC m=+1399.958296626" watchObservedRunningTime="2026-10-31 03:39:35.78339434 +0000 UTC m=+1399.977917463" Oct 31 03:39:35.809593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.809540 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" podStartSLOduration=18.809525455 podStartE2EDuration="18.809525455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:35.789081184 +0000 UTC m=+1399.983604306" watchObservedRunningTime="2026-10-31 03:39:35.809525455 +0000 UTC m=+1400.004048575" Oct 31 03:39:35.812307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.812278 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb"] Oct 31 03:39:35.815146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:35.815120 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adb7b7-67649788bc-qlhjb"] Oct 31 03:39:36.041495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.041411 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50bb7aaf-5529-4cd5-ace6-1f8e89b243c7" path="/var/lib/kubelet/pods/50bb7aaf-5529-4cd5-ace6-1f8e89b243c7/volumes" Oct 31 03:39:36.048377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.042510 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891e5f8d-1360-406b-894d-8cb637655611" path="/var/lib/kubelet/pods/891e5f8d-1360-406b-894d-8cb637655611/volumes" Oct 31 03:39:36.048377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.043088 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6143c03-8a03-4e2b-ac02-18ac593c103f" path="/var/lib/kubelet/pods/c6143c03-8a03-4e2b-ac02-18ac593c103f/volumes" Oct 31 03:39:36.048377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.043647 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb96d9d-566b-4d00-9c59-857a2d53552f" path="/var/lib/kubelet/pods/fdb96d9d-566b-4d00-9c59-857a2d53552f/volumes" Oct 31 03:39:36.302771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.302647 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b"] Oct 31 03:39:36.322919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.322888 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c6143c03-8a03-4e2b-ac02-18ac593c103f" containerName="payload-pre-processing" Oct 31 03:39:36.323144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.323128 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6143c03-8a03-4e2b-ac02-18ac593c103f" containerName="payload-pre-processing" Oct 31 03:39:36.323373 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.323357 5520 cpu_manager.go:414] "containerMap: removing container" podUID="891e5f8d-1360-406b-894d-8cb637655611" containerName="payload-processing" Oct 31 03:39:36.329273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.329240 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891e5f8d-1360-406b-894d-8cb637655611" containerName="payload-processing" Oct 31 03:39:36.329534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.329519 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6143c03-8a03-4e2b-ac02-18ac593c103f" containerName="payload-pre-processing" Oct 31 03:39:36.329623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.329551 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="891e5f8d-1360-406b-894d-8cb637655611" containerName="payload-processing" Oct 31 03:39:36.338623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.338597 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b"] Oct 31 03:39:36.338917 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.338859 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.347062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.347034 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8385c5\"/\"openshift-service-ca.crt\"" Oct 31 03:39:36.347195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.347167 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8385c5\"/\"default-dockercfg-zgq2z\"" Oct 31 03:39:36.347195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.347230 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8385c5\"/\"kube-root-ca.crt\"" Oct 31 03:39:36.347376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.347317 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8385c5\"/\"test-model-8385c5-kserve-self-signed-certs\"" Oct 31 03:39:36.427058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.427020 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-model-cache\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.427318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.427147 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tmp-dir\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.427318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.427238 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2ph\" (UniqueName: \"kubernetes.io/projected/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-kube-api-access-wm2ph\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.427318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.427279 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-home\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.427516 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.427407 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tls-certs\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.427516 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.427462 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-dshm\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.529599 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tls-certs\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.529883 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-dshm\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.529999 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-model-cache\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.530218 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tmp-dir\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.530250 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2ph\" (UniqueName: \"kubernetes.io/projected/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-kube-api-access-wm2ph\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.530301 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-home\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.531159 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-home\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.531897 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-model-cache\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.532788 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tmp-dir\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.540933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.540902 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-dshm\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.574861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.554655 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tls-certs\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.574861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.570432 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2ph\" (UniqueName: \"kubernetes.io/projected/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-kube-api-access-wm2ph\") pod \"test-model-8385c5-kserve-87c4884d7-gx92b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:36.674555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:36.674520 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:37.297545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:37.296390 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b"] Oct 31 03:39:37.779653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:37.779609 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" event={"ID":"79b1b755-4ff4-4e73-99f9-fb3f52fb415b","Type":"ContainerStarted","Data":"f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e"} Oct 31 03:39:37.779903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:37.779664 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" event={"ID":"79b1b755-4ff4-4e73-99f9-fb3f52fb415b","Type":"ContainerStarted","Data":"1fccee48dc9025dfdef127996e74bf070fc460cdfb728123cb5079199d796583"} Oct 31 03:39:37.800424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:37.800360 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podStartSLOduration=1.800340865 podStartE2EDuration="1.800340865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:37.796438558 +0000 UTC m=+1401.990961680" watchObservedRunningTime="2026-10-31 03:39:37.800340865 +0000 UTC m=+1401.994863989" Oct 31 03:39:38.674982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:38.674933 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:39.362705 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:39.362654 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:39:46.675231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:46.675170 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:46.688570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:46.688545 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:46.773336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:46.773306 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:39:46.775248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:46.774806 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:39:46.953565 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:46.953538 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:39:48.389899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.389863 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k"] Oct 31 03:39:48.402342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.400520 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k"] Oct 31 03:39:48.402618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.400675 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.408324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.408008 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f28b4\"/\"default-dockercfg-7qmff\"" Oct 31 03:39:48.408324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.408101 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6f28b4\"/\"test-model-6f28b4-kserve-self-signed-certs\"" Oct 31 03:39:48.408324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.408196 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f28b4\"/\"kube-root-ca.crt\"" Oct 31 03:39:48.408324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.408293 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6f28b4\"/\"openshift-service-ca.crt\"" Oct 31 03:39:48.512151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.512113 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-model-cache\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.512151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.512158 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-home\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.512455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.512220 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-tmp-dir\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.512455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.512432 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-dshm\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.512587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.512505 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwk4t\" (UniqueName: \"kubernetes.io/projected/759215f2-2e3f-4dbd-9009-166bc4e3819d-kube-api-access-qwk4t\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.512587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.512546 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759215f2-2e3f-4dbd-9009-166bc4e3819d-tls-certs\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.617565 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.617524 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-dshm\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.618310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.618282 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwk4t\" (UniqueName: \"kubernetes.io/projected/759215f2-2e3f-4dbd-9009-166bc4e3819d-kube-api-access-qwk4t\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.618512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.618497 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759215f2-2e3f-4dbd-9009-166bc4e3819d-tls-certs\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.622529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.622499 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-model-cache\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.649275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.648727 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-home\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.649275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.648830 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-tmp-dir\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.649492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.649378 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-tmp-dir\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.649492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.626731 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-model-cache\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.653224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.649711 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-home\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.657224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.655770 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759215f2-2e3f-4dbd-9009-166bc4e3819d-tls-certs\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.675223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.665974 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-dshm\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.685225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.676493 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwk4t\" (UniqueName: \"kubernetes.io/projected/759215f2-2e3f-4dbd-9009-166bc4e3819d-kube-api-access-qwk4t\") pod \"test-model-6f28b4-kserve-6c486dc6bc-qzq5k\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:48.734230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:48.733573 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:49.450733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:49.450704 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k"] Oct 31 03:39:49.451845 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:39:49.451821 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod759215f2_2e3f_4dbd_9009_166bc4e3819d.slice/crio-e4ddd89fa117af43b464e37a2457d48d832390df2e0761fb292848ef55fe3d9d WatchSource:0}: Error finding container e4ddd89fa117af43b464e37a2457d48d832390df2e0761fb292848ef55fe3d9d: Status 404 returned error can't find the container with id e4ddd89fa117af43b464e37a2457d48d832390df2e0761fb292848ef55fe3d9d Oct 31 03:39:50.055602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:50.055562 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" event={"ID":"759215f2-2e3f-4dbd-9009-166bc4e3819d","Type":"ContainerStarted","Data":"a6c297918b7b485342a2c3d6bc0a25669f9fa9b1fe86a2332845b167d2421460"} Oct 31 03:39:50.055811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:50.055617 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" event={"ID":"759215f2-2e3f-4dbd-9009-166bc4e3819d","Type":"ContainerStarted","Data":"e4ddd89fa117af43b464e37a2457d48d832390df2e0761fb292848ef55fe3d9d"} Oct 31 03:39:50.077557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:50.077507 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podStartSLOduration=2.077491912 podStartE2EDuration="2.077491912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:50.071680101 +0000 UTC m=+1414.266203250" watchObservedRunningTime="2026-10-31 03:39:50.077491912 +0000 UTC m=+1414.272015060" Oct 31 03:39:50.734405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:50.734372 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:51.420434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:51.420354 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw"] Oct 31 03:39:51.420652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:51.420626 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" podUID="fd1731d4-7cce-41f3-945f-ffded1fad0a8" containerName="maas-api" containerID="cri-o://f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b" gracePeriod=30 Oct 31 03:39:51.605425 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:51.605367 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9"] Oct 31 03:39:51.613368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:51.613307 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-555bd6-x5fq9"] Oct 31 03:39:52.082226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.065441 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085a7ddc-b35e-41c7-86cf-b594d25c7703" path="/var/lib/kubelet/pods/085a7ddc-b35e-41c7-86cf-b594d25c7703/volumes" Oct 31 03:39:52.084717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.084292 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:39:52.198228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.186655 5520 generic.go:356] "Generic (PLEG): container finished" podID="fd1731d4-7cce-41f3-945f-ffded1fad0a8" containerID="f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b" exitCode=0 Oct 31 03:39:52.198228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.188106 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" Oct 31 03:39:52.198228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.188558 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" event={"ID":"fd1731d4-7cce-41f3-945f-ffded1fad0a8","Type":"ContainerDied","Data":"f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b"} Oct 31 03:39:52.198228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.188607 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw" event={"ID":"fd1731d4-7cce-41f3-945f-ffded1fad0a8","Type":"ContainerDied","Data":"bce39fba7e75d1bf31281466a213de8196338eb9654a8c7c87de846bead04cb2"} Oct 31 03:39:52.198228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.188639 5520 scope.go:122] "RemoveContainer" containerID="f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b" Oct 31 03:39:52.336611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.336577 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh"] Oct 31 03:39:52.340858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.337964 5520 cpu_manager.go:414] "containerMap: removing container" podUID="fd1731d4-7cce-41f3-945f-ffded1fad0a8" containerName="maas-api" Oct 31 03:39:52.340858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.337989 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1731d4-7cce-41f3-945f-ffded1fad0a8" containerName="maas-api" Oct 31 03:39:52.340858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.338189 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1731d4-7cce-41f3-945f-ffded1fad0a8" containerName="maas-api" Oct 31 03:39:52.346720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.346663 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.348858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.348834 5520 scope.go:122] "RemoveContainer" containerID="f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b" Oct 31 03:39:52.349322 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:39:52.349264 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b\": container with ID starting with f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b not found: ID does not exist" containerID="f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b" Oct 31 03:39:52.349322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.349298 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b"} err="failed to get container status \"f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b\": rpc error: code = NotFound desc = could not find container \"f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b\": container with ID starting with f35a993d36fb425ec483b4718915ccfa3425a170f753eb8cb517ace73f9ea52b not found: ID does not exist" Oct 31 03:39:52.351745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.351720 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-880e04-openshift-default-dockercfg-8z2d9\"" Oct 31 03:39:52.362546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.362515 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh"] Oct 31 03:39:52.372181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.372141 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-maas-api-tls\") pod \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " Oct 31 03:39:52.372471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.372449 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1731d4-7cce-41f3-945f-ffded1fad0a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1731d4-7cce-41f3-945f-ffded1fad0a8-openshift-service-ca\") pod \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " Oct 31 03:39:52.372563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.372547 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1731d4-7cce-41f3-945f-ffded1fad0a8-kube-api-access-bfhvp\" (UniqueName: \"kubernetes.io/projected/fd1731d4-7cce-41f3-945f-ffded1fad0a8-kube-api-access-bfhvp\") pod \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " Oct 31 03:39:52.372700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.372685 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-metrics-certs\") pod \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\" (UID: \"fd1731d4-7cce-41f3-945f-ffded1fad0a8\") " Oct 31 03:39:52.374342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.374315 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1731d4-7cce-41f3-945f-ffded1fad0a8-openshift-service-ca" pod "fd1731d4-7cce-41f3-945f-ffded1fad0a8" (UID: "fd1731d4-7cce-41f3-945f-ffded1fad0a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:39:52.375763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.375736 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-metrics-certs" pod "fd1731d4-7cce-41f3-945f-ffded1fad0a8" (UID: "fd1731d4-7cce-41f3-945f-ffded1fad0a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:52.376137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.376101 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-maas-api-tls" pod "fd1731d4-7cce-41f3-945f-ffded1fad0a8" (UID: "fd1731d4-7cce-41f3-945f-ffded1fad0a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:39:52.376619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.376593 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1731d4-7cce-41f3-945f-ffded1fad0a8-kube-api-access-bfhvp" pod "fd1731d4-7cce-41f3-945f-ffded1fad0a8" (UID: "fd1731d4-7cce-41f3-945f-ffded1fad0a8"). InnerVolumeSpecName "kube-api-access-bfhvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:39:52.474661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.474622 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-podinfo\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.474847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.474754 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.474847 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.474839 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istiod-ca-cert\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.474974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.474915 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-token\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.474974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.474938 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-credential-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.475048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.474975 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-certs\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.475103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.475090 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-data\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.475144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.475115 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzv8x\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-kube-api-access-wzv8x\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.475329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.475311 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-envoy\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.475504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.475481 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:52.475570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.475514 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1731d4-7cce-41f3-945f-ffded1fad0a8-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:52.475570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.475534 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfhvp\" (UniqueName: \"kubernetes.io/projected/fd1731d4-7cce-41f3-945f-ffded1fad0a8-kube-api-access-bfhvp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:52.475570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.475552 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1731d4-7cce-41f3-945f-ffded1fad0a8-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:39:52.516522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.516477 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw"] Oct 31 03:39:52.521978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.521951 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-555bd6-68dbbf8f94-7b7jw"] Oct 31 03:39:52.576705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.576658 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-podinfo\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.576884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.576790 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.576950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.576912 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istiod-ca-cert\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.576996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.576980 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-token\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577017 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-credential-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577074 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-certs\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577169 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-data\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577250 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv8x\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-kube-api-access-wzv8x\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577324 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577395 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-envoy\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577486 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-credential-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.577947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.577918 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-data\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.578297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.578274 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istiod-ca-cert\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.578419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.578301 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-certs\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.579947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.579924 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-podinfo\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.580517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.580498 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-envoy\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.585641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.585619 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-token\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.586096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.586075 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv8x\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-kube-api-access-wzv8x\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-7plsh\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.670024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.669986 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:52.990570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.990542 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh"] Oct 31 03:39:52.992295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.992263 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:52.992383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.992345 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:52.992434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:52.992405 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:39:53.200032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:53.199993 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" event={"ID":"929fc3a7-eaf6-4d34-8754-dec95c4aff02","Type":"ContainerStarted","Data":"a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7"} Oct 31 03:39:53.200558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:53.200059 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" event={"ID":"929fc3a7-eaf6-4d34-8754-dec95c4aff02","Type":"ContainerStarted","Data":"bfe757b5bb638da9bb756b19bf868b48667026d2b585370a8b16997cbcf8798b"} Oct 31 03:39:53.228695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:53.228632 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podStartSLOduration=29.228613252 podStartE2EDuration="29.228613252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:53.221956948 +0000 UTC m=+1417.416480070" watchObservedRunningTime="2026-10-31 03:39:53.228613252 +0000 UTC m=+1417.423136374" Oct 31 03:39:53.690124 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:53.689310 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:53.924075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:53.924039 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1731d4-7cce-41f3-945f-ffded1fad0a8" path="/var/lib/kubelet/pods/fd1731d4-7cce-41f3-945f-ffded1fad0a8/volumes" Oct 31 03:39:54.677834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:54.677802 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:55.290824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:55.290788 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:55.291978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:55.291952 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:39:56.794565 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.794525 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l"] Oct 31 03:39:56.801973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.801947 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:56.806384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.805804 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-880e04-dockercfg-rzsbn\"" Oct 31 03:39:56.806828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.806791 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-880e04\"" Oct 31 03:39:56.810868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.810841 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l"] Oct 31 03:39:56.824831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.824802 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8"] Oct 31 03:39:56.858615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.858582 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:56.860713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.860685 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8"] Oct 31 03:39:56.912956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.912922 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gfgz\" (UniqueName: \"kubernetes.io/projected/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-kube-api-access-9gfgz\") pod \"payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:56.913266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:56.913244 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:57.028662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.028616 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gfgz\" (UniqueName: \"kubernetes.io/projected/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-kube-api-access-9gfgz\") pod \"payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:57.030265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.030233 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004fe8fb-5074-440f-b43c-f95f17fdbba0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:57.031106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.031082 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzgq\" (UniqueName: \"kubernetes.io/projected/004fe8fb-5074-440f-b43c-f95f17fdbba0-kube-api-access-8kzgq\") pod \"payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:57.031380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.031354 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:57.036227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.032336 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:57.055505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.055430 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gfgz\" (UniqueName: \"kubernetes.io/projected/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-kube-api-access-9gfgz\") pod \"payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:57.147094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.147043 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004fe8fb-5074-440f-b43c-f95f17fdbba0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:57.148861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.148836 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzgq\" (UniqueName: \"kubernetes.io/projected/004fe8fb-5074-440f-b43c-f95f17fdbba0-kube-api-access-8kzgq\") pod \"payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:57.149523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.148591 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004fe8fb-5074-440f-b43c-f95f17fdbba0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:57.161096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.161070 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:57.167870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.167756 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzgq\" (UniqueName: \"kubernetes.io/projected/004fe8fb-5074-440f-b43c-f95f17fdbba0-kube-api-access-8kzgq\") pod \"payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:57.187540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.187508 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:57.727901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.727120 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8"] Oct 31 03:39:57.818315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:57.815920 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l"] Oct 31 03:39:57.831449 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:39:57.831420 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc6d6f3_7ba7_4e5d_9e97_219354a72025.slice/crio-29512d8a7915a7e54476e90cdd4cd7adc3cb38e946465567b80c2c97697115b2 WatchSource:0}: Error finding container 29512d8a7915a7e54476e90cdd4cd7adc3cb38e946465567b80c2c97697115b2: Status 404 returned error can't find the container with id 29512d8a7915a7e54476e90cdd4cd7adc3cb38e946465567b80c2c97697115b2 Oct 31 03:39:58.383292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.383148 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" event={"ID":"004fe8fb-5074-440f-b43c-f95f17fdbba0","Type":"ContainerStarted","Data":"33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821"} Oct 31 03:39:58.383292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.383240 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" event={"ID":"004fe8fb-5074-440f-b43c-f95f17fdbba0","Type":"ContainerStarted","Data":"6be7574bc26b3def790fc10c89739810cc97f2e9cd9638323e613b30d4bef4d9"} Oct 31 03:39:58.383292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.383288 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:39:58.385732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.385702 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" event={"ID":"6fc6d6f3-7ba7-4e5d-9e97-219354a72025","Type":"ContainerStarted","Data":"4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f"} Oct 31 03:39:58.385876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.385753 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" event={"ID":"6fc6d6f3-7ba7-4e5d-9e97-219354a72025","Type":"ContainerStarted","Data":"29512d8a7915a7e54476e90cdd4cd7adc3cb38e946465567b80c2c97697115b2"} Oct 31 03:39:58.385876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.385809 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:39:58.406045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.405992 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" podStartSLOduration=2.405974489 podStartE2EDuration="2.405974489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:58.400046392 +0000 UTC m=+1422.594569516" watchObservedRunningTime="2026-10-31 03:39:58.405974489 +0000 UTC m=+1422.600497611" Oct 31 03:39:58.418398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.418356 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" podStartSLOduration=2.4183419 podStartE2EDuration="2.4183419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:58.416681648 +0000 UTC m=+1422.611204769" watchObservedRunningTime="2026-10-31 03:39:58.4183419 +0000 UTC m=+1422.612865021" Oct 31 03:39:58.735806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.735779 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:58.749450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:58.749425 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:39:59.428230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:39:59.424389 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:40:06.467939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:06.467040 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:09.360614 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:09.360570 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:40:09.401835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:09.401809 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:40:09.402017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:09.401880 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:40:10.117685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:10.117650 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f"] Oct 31 03:40:10.133234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:10.133181 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8"] Oct 31 03:40:11.934688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:11.934654 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8"] Oct 31 03:40:11.935234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:11.934950 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" podUID="75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" containerName="maas-api" containerID="cri-o://5f7b00b85aa0bbf1e282bc7af02986d72d8c7670cf943ef43863417db26fadd3" gracePeriod=30 Oct 31 03:40:12.748171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.745960 5520 generic.go:356] "Generic (PLEG): container finished" podID="75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" containerID="5f7b00b85aa0bbf1e282bc7af02986d72d8c7670cf943ef43863417db26fadd3" exitCode=0 Oct 31 03:40:12.748171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.746243 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" event={"ID":"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d","Type":"ContainerDied","Data":"5f7b00b85aa0bbf1e282bc7af02986d72d8c7670cf943ef43863417db26fadd3"} Oct 31 03:40:12.828733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.828580 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:40:12.897229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.896918 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp"] Oct 31 03:40:12.897878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.897855 5520 cpu_manager.go:414] "containerMap: removing container" podUID="75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" containerName="maas-api" Oct 31 03:40:12.897878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.897876 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" containerName="maas-api" Oct 31 03:40:12.898086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.898076 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" containerName="maas-api" Oct 31 03:40:12.902589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.902569 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:12.911301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.911274 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp"] Oct 31 03:40:12.926567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.926531 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-maas-api-tls\") pod \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " Oct 31 03:40:12.926728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.926607 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-openshift-service-ca\") pod \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " Oct 31 03:40:12.926894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.926875 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-metrics-certs\") pod \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " Oct 31 03:40:12.926965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.926925 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-kube-api-access-xgpbw\" (UniqueName: \"kubernetes.io/projected/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-kube-api-access-xgpbw\") pod \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\" (UID: \"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d\") " Oct 31 03:40:12.935905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.935874 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-openshift-service-ca" pod "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" (UID: "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:12.948260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.943669 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-maas-api-tls" pod "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" (UID: "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:12.948260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.943754 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-kube-api-access-xgpbw" pod "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" (UID: "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d"). InnerVolumeSpecName "kube-api-access-xgpbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:12.950069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.950042 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-metrics-certs" pod "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" (UID: "75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:12.964141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:12.964121 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a3b986-serving-cert\"" Oct 31 03:40:13.033860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.033820 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72m9\" (UniqueName: \"kubernetes.io/projected/9cf77425-e97a-48e6-8d80-751e612ee153-kube-api-access-w72m9\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.034056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.033955 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-metrics-certs\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.034479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.034390 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-maas-api-tls\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.034623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.034555 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf77425-e97a-48e6-8d80-751e612ee153-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.034853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.034835 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:13.034931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.034860 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpbw\" (UniqueName: \"kubernetes.io/projected/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-kube-api-access-xgpbw\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:13.034931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.034877 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:13.034931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.034892 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:13.136823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.136786 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72m9\" (UniqueName: \"kubernetes.io/projected/9cf77425-e97a-48e6-8d80-751e612ee153-kube-api-access-w72m9\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.137018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.136856 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-metrics-certs\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.137018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.136982 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-maas-api-tls\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.137138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.137027 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf77425-e97a-48e6-8d80-751e612ee153-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.137628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.137602 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf77425-e97a-48e6-8d80-751e612ee153-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.139410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.139386 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-metrics-certs\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.139583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.139561 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-maas-api-tls\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.147024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.146998 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72m9\" (UniqueName: \"kubernetes.io/projected/9cf77425-e97a-48e6-8d80-751e612ee153-kube-api-access-w72m9\") pod \"maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.237006 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.236971 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:13.561108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.561083 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp"] Oct 31 03:40:13.561265 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:40:13.561077 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf77425_e97a_48e6_8d80_751e612ee153.slice/crio-068c4bc395cb82dad5f9d97c4fe066ab3461e2be6ed8e617fb2d15f2e8d39013 WatchSource:0}: Error finding container 068c4bc395cb82dad5f9d97c4fe066ab3461e2be6ed8e617fb2d15f2e8d39013: Status 404 returned error can't find the container with id 068c4bc395cb82dad5f9d97c4fe066ab3461e2be6ed8e617fb2d15f2e8d39013 Oct 31 03:40:13.809278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.808319 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" event={"ID":"75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d","Type":"ContainerDied","Data":"942aaa732e8e67546c30ed0a655383ebe8f501fdec8cce6ecc4d1ce766be7805"} Oct 31 03:40:13.809278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.808398 5520 scope.go:122] "RemoveContainer" containerID="5f7b00b85aa0bbf1e282bc7af02986d72d8c7670cf943ef43863417db26fadd3" Oct 31 03:40:13.809278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.808598 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8" Oct 31 03:40:13.815123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.815052 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" event={"ID":"9cf77425-e97a-48e6-8d80-751e612ee153","Type":"ContainerStarted","Data":"068c4bc395cb82dad5f9d97c4fe066ab3461e2be6ed8e617fb2d15f2e8d39013"} Oct 31 03:40:13.852068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.852016 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8"] Oct 31 03:40:13.857924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.857893 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adb7b7-65bf574688-kmjk8"] Oct 31 03:40:13.922979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:13.922941 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d" path="/var/lib/kubelet/pods/75e26c2c-cf0c-4cf5-bf11-b3f31a33ba1d/volumes" Oct 31 03:40:14.829248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:14.826951 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" event={"ID":"9cf77425-e97a-48e6-8d80-751e612ee153","Type":"ContainerStarted","Data":"d3e945e4f2a1860a7fb1d7b143e5b585d3ba692cae2c63d3de5fa91548a76b3b"} Oct 31 03:40:14.829248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:14.827027 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:14.848555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:14.848483 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" podStartSLOduration=56.369259196 podStartE2EDuration="56.848464198s" totalImagesPullingTime="479.205002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:14.843968268 +0000 UTC m=+1439.038491408" watchObservedRunningTime="2026-10-31 03:40:14.848464198 +0000 UTC m=+1439.042987328" Oct 31 03:40:15.118912 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.118810 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" podUID="2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" containerName="payload-processing" containerID="cri-o://d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9" gracePeriod=25 Oct 31 03:40:15.134083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.134049 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" podUID="ff74a954-22a6-4353-a2e5-0ac91d6137a0" containerName="payload-pre-processing" containerID="cri-o://9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c" gracePeriod=25 Oct 31 03:40:15.582186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.582161 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:40:15.627306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.627278 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:40:15.691528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.691482 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-kube-api-access-b5vtn\" (UniqueName: \"kubernetes.io/projected/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-kube-api-access-b5vtn\") pod \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " Oct 31 03:40:15.691709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.691544 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-plugins-config-volume\") pod \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\" (UID: \"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8\") " Oct 31 03:40:15.691948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.691922 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-plugins-config-volume" pod "2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" (UID: "2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:15.692125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.692113 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:15.694009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.693980 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-kube-api-access-b5vtn" pod "2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" (UID: "2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8"). InnerVolumeSpecName "kube-api-access-b5vtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:15.793263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.793163 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff74a954-22a6-4353-a2e5-0ac91d6137a0-kube-api-access-rw6s2\" (UniqueName: \"kubernetes.io/projected/ff74a954-22a6-4353-a2e5-0ac91d6137a0-kube-api-access-rw6s2\") pod \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " Oct 31 03:40:15.793424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.793388 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff74a954-22a6-4353-a2e5-0ac91d6137a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff74a954-22a6-4353-a2e5-0ac91d6137a0-plugins-config-volume\") pod \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\" (UID: \"ff74a954-22a6-4353-a2e5-0ac91d6137a0\") " Oct 31 03:40:15.793769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.793741 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff74a954-22a6-4353-a2e5-0ac91d6137a0-plugins-config-volume" pod "ff74a954-22a6-4353-a2e5-0ac91d6137a0" (UID: "ff74a954-22a6-4353-a2e5-0ac91d6137a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:15.794255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.794237 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff74a954-22a6-4353-a2e5-0ac91d6137a0-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:15.794315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.794260 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5vtn\" (UniqueName: \"kubernetes.io/projected/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8-kube-api-access-b5vtn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:15.795792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.795755 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff74a954-22a6-4353-a2e5-0ac91d6137a0-kube-api-access-rw6s2" pod "ff74a954-22a6-4353-a2e5-0ac91d6137a0" (UID: "ff74a954-22a6-4353-a2e5-0ac91d6137a0"). InnerVolumeSpecName "kube-api-access-rw6s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:15.838832 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.838798 5520 generic.go:356] "Generic (PLEG): container finished" podID="2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" containerID="d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9" exitCode=0 Oct 31 03:40:15.839365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.838849 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" Oct 31 03:40:15.839365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.838885 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" event={"ID":"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8","Type":"ContainerDied","Data":"d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9"} Oct 31 03:40:15.839365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.838956 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f" event={"ID":"2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8","Type":"ContainerDied","Data":"320b6e68db5fa77fc4893626e9d6d693fc02903b38f721d68d207a1bd7cabf8b"} Oct 31 03:40:15.839365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.838993 5520 scope.go:122] "RemoveContainer" containerID="d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9" Oct 31 03:40:15.840954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.840904 5520 generic.go:356] "Generic (PLEG): container finished" podID="ff74a954-22a6-4353-a2e5-0ac91d6137a0" containerID="9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c" exitCode=0 Oct 31 03:40:15.841070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.840968 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" Oct 31 03:40:15.841070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.840985 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" event={"ID":"ff74a954-22a6-4353-a2e5-0ac91d6137a0","Type":"ContainerDied","Data":"9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c"} Oct 31 03:40:15.841070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.841035 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8" event={"ID":"ff74a954-22a6-4353-a2e5-0ac91d6137a0","Type":"ContainerDied","Data":"9a9e8d314b03b1e03149018e9dabc51bc1b1facd90ad5fd017f31c390d362e0d"} Oct 31 03:40:15.854134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.854111 5520 scope.go:122] "RemoveContainer" containerID="d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9" Oct 31 03:40:15.854420 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:15.854400 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9\": container with ID starting with d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9 not found: ID does not exist" containerID="d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9" Oct 31 03:40:15.854520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.854427 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9"} err="failed to get container status \"d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9\": rpc error: code = NotFound desc = could not find container \"d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9\": container with ID starting with d1224a79f6da018837f399eb83d47d9d78aa666854ddb74eec3d5418309b4cf9 not found: ID does not exist" Oct 31 03:40:15.854520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.854443 5520 scope.go:122] "RemoveContainer" containerID="9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c" Oct 31 03:40:15.866296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.866260 5520 scope.go:122] "RemoveContainer" containerID="9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c" Oct 31 03:40:15.866650 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:15.866629 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c\": container with ID starting with 9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c not found: ID does not exist" containerID="9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c" Oct 31 03:40:15.866766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.866656 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c"} err="failed to get container status \"9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c\": rpc error: code = NotFound desc = could not find container \"9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c\": container with ID starting with 9b51197f44191cebb69ed2236784af6b274b81a9603d087a3caa752334dce46c not found: ID does not exist" Oct 31 03:40:15.868947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.868928 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f"] Oct 31 03:40:15.875982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.875769 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8361df-79785f96df-fcf5f"] Oct 31 03:40:15.880886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.880855 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8"] Oct 31 03:40:15.884921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.884903 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8361df-bf4c79d87-v4ct8"] Oct 31 03:40:15.895509 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.895489 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw6s2\" (UniqueName: \"kubernetes.io/projected/ff74a954-22a6-4353-a2e5-0ac91d6137a0-kube-api-access-rw6s2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:15.921949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.921921 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" path="/var/lib/kubelet/pods/2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8/volumes" Oct 31 03:40:15.922550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:15.922532 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff74a954-22a6-4353-a2e5-0ac91d6137a0" path="/var/lib/kubelet/pods/ff74a954-22a6-4353-a2e5-0ac91d6137a0/volumes" Oct 31 03:40:20.846934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:20.846903 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:40:31.695925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:31.695853 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr"] Oct 31 03:40:36.696177 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:36.696130 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" podUID="300883b1-3b94-4ff2-88bd-668000fa70f7" containerName="payload-processing" containerID="cri-o://0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a" gracePeriod=25 Oct 31 03:40:36.755707 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:36.755665 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300883b1_3b94_4ff2_88bd_668000fa70f7.slice/crio-0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:40:37.025718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.025691 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:40:37.113949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.112217 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7"] Oct 31 03:40:37.114612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.114584 5520 cpu_manager.go:414] "containerMap: removing container" podUID="300883b1-3b94-4ff2-88bd-668000fa70f7" containerName="payload-processing" Oct 31 03:40:37.114819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.114794 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300883b1-3b94-4ff2-88bd-668000fa70f7" containerName="payload-processing" Oct 31 03:40:37.114934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.114885 5520 cpu_manager.go:414] "containerMap: removing container" podUID="2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" containerName="payload-processing" Oct 31 03:40:37.114934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.114895 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" containerName="payload-processing" Oct 31 03:40:37.114934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.114931 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ff74a954-22a6-4353-a2e5-0ac91d6137a0" containerName="payload-pre-processing" Oct 31 03:40:37.115108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.114940 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff74a954-22a6-4353-a2e5-0ac91d6137a0" containerName="payload-pre-processing" Oct 31 03:40:37.115428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.115409 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="300883b1-3b94-4ff2-88bd-668000fa70f7" containerName="payload-processing" Oct 31 03:40:37.115506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.115454 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aab5c5f-a03f-4f52-bc53-a21dfc8ec8e8" containerName="payload-processing" Oct 31 03:40:37.115506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.115463 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff74a954-22a6-4353-a2e5-0ac91d6137a0" containerName="payload-pre-processing" Oct 31 03:40:37.124835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.124803 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7"] Oct 31 03:40:37.124941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.124918 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.128786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.128765 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-73e222-openshift-default-dockercfg-l6dr2\"" Oct 31 03:40:37.142381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.142361 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300883b1-3b94-4ff2-88bd-668000fa70f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300883b1-3b94-4ff2-88bd-668000fa70f7-plugins-config-volume\") pod \"300883b1-3b94-4ff2-88bd-668000fa70f7\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " Oct 31 03:40:37.142513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.142499 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300883b1-3b94-4ff2-88bd-668000fa70f7-kube-api-access-8hbb5\" (UniqueName: \"kubernetes.io/projected/300883b1-3b94-4ff2-88bd-668000fa70f7-kube-api-access-8hbb5\") pod \"300883b1-3b94-4ff2-88bd-668000fa70f7\" (UID: \"300883b1-3b94-4ff2-88bd-668000fa70f7\") " Oct 31 03:40:37.142710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.142691 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300883b1-3b94-4ff2-88bd-668000fa70f7-plugins-config-volume" pod "300883b1-3b94-4ff2-88bd-668000fa70f7" (UID: "300883b1-3b94-4ff2-88bd-668000fa70f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:37.143343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.143273 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/300883b1-3b94-4ff2-88bd-668000fa70f7-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:37.144819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.144798 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300883b1-3b94-4ff2-88bd-668000fa70f7-kube-api-access-8hbb5" pod "300883b1-3b94-4ff2-88bd-668000fa70f7" (UID: "300883b1-3b94-4ff2-88bd-668000fa70f7"). InnerVolumeSpecName "kube-api-access-8hbb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:37.245383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245305 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-certs\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245372 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-envoy\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245509 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8c7\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-kube-api-access-cp8c7\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245566 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245583 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-credential-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245664 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3f882-054f-4e82-b27e-1a0c6cfda587-istiod-ca-cert\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245753 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-token\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245845 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-data\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.245893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.245882 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-podinfo\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.246057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.246046 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbb5\" (UniqueName: \"kubernetes.io/projected/300883b1-3b94-4ff2-88bd-668000fa70f7-kube-api-access-8hbb5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354506 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-certs\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354594 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-envoy\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354718 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8c7\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-kube-api-access-cp8c7\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354787 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354810 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-credential-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354835 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3f882-054f-4e82-b27e-1a0c6cfda587-istiod-ca-cert\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354870 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-token\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.354949 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-data\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.355003 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-podinfo\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.355813 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.356233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.356054 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-certs\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.382228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.380757 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3f882-054f-4e82-b27e-1a0c6cfda587-istiod-ca-cert\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.382228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.381021 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-credential-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.382228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.381549 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-data\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.392295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.391057 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-podinfo\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.405170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.403343 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-envoy\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.405800 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8c7\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-kube-api-access-cp8c7\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.407958 5520 generic.go:356] "Generic (PLEG): container finished" podID="300883b1-3b94-4ff2-88bd-668000fa70f7" containerID="0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a" exitCode=0 Oct 31 03:40:37.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.408025 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" Oct 31 03:40:37.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.408085 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" event={"ID":"300883b1-3b94-4ff2-88bd-668000fa70f7","Type":"ContainerDied","Data":"0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a"} Oct 31 03:40:37.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.408128 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr" event={"ID":"300883b1-3b94-4ff2-88bd-668000fa70f7","Type":"ContainerDied","Data":"bab63399a45f7ac19cc23fa50ce5eb4581cb6051725130ee8eba93bee4bae20d"} Oct 31 03:40:37.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.408154 5520 scope.go:122] "RemoveContainer" containerID="0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a" Oct 31 03:40:37.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.410859 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-token\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.437312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.437277 5520 scope.go:122] "RemoveContainer" containerID="0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a" Oct 31 03:40:37.438286 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:37.438195 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a\": container with ID starting with 0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a not found: ID does not exist" containerID="0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a" Oct 31 03:40:37.438286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.438253 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a"} err="failed to get container status \"0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a\": rpc error: code = NotFound desc = could not find container \"0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a\": container with ID starting with 0ef27bf0e3ecfb68e92afc363749ecb92377934b3519f48ea6440010199e304a not found: ID does not exist" Oct 31 03:40:37.445311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.445286 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:37.448399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.448377 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr"] Oct 31 03:40:37.463760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.463734 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-619d46-98d6f8b6f-vnkpr"] Oct 31 03:40:37.761008 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:40:37.760961 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d3f882_054f_4e82_b27e_1a0c6cfda587.slice/crio-60043e45588fa050d0fc872c0b37cc2ad4132c3b0e3a00de1f9d3826281c5a90 WatchSource:0}: Error finding container 60043e45588fa050d0fc872c0b37cc2ad4132c3b0e3a00de1f9d3826281c5a90: Status 404 returned error can't find the container with id 60043e45588fa050d0fc872c0b37cc2ad4132c3b0e3a00de1f9d3826281c5a90 Oct 31 03:40:37.761706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.761484 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7"] Oct 31 03:40:37.764069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.763999 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:40:37.764170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.764102 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:40:37.764170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.764138 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:40:37.925474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:37.925441 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300883b1-3b94-4ff2-88bd-668000fa70f7" path="/var/lib/kubelet/pods/300883b1-3b94-4ff2-88bd-668000fa70f7/volumes" Oct 31 03:40:38.516237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:38.514000 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" event={"ID":"82d3f882-054f-4e82-b27e-1a0c6cfda587","Type":"ContainerStarted","Data":"4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b"} Oct 31 03:40:38.516237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:38.514076 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" event={"ID":"82d3f882-054f-4e82-b27e-1a0c6cfda587","Type":"ContainerStarted","Data":"60043e45588fa050d0fc872c0b37cc2ad4132c3b0e3a00de1f9d3826281c5a90"} Oct 31 03:40:38.610227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:38.593291 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podStartSLOduration=10.593270419 podStartE2EDuration="10.593270419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:38.54769279 +0000 UTC m=+1462.742215937" watchObservedRunningTime="2026-10-31 03:40:38.593270419 +0000 UTC m=+1462.787793542" Oct 31 03:40:39.360862 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:39.360822 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:40:39.445868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:39.445836 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:39.453626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:39.453596 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:39.527399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:39.527327 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:39.528682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:39.528660 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:40:41.369393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:41.369356 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb"] Oct 31 03:40:46.370703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:46.370647 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" podUID="74235b76-3bed-49e7-98ae-a300f7be917f" containerName="payload-processing" containerID="cri-o://386191c08999d1a33f76eab1bbe5ff8deb414396784555accb52179c1b03d2f2" gracePeriod=25 Oct 31 03:40:46.691456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:46.691421 5520 generic.go:356] "Generic (PLEG): container finished" podID="74235b76-3bed-49e7-98ae-a300f7be917f" containerID="386191c08999d1a33f76eab1bbe5ff8deb414396784555accb52179c1b03d2f2" exitCode=0 Oct 31 03:40:46.694036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:46.694004 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" event={"ID":"74235b76-3bed-49e7-98ae-a300f7be917f","Type":"ContainerDied","Data":"386191c08999d1a33f76eab1bbe5ff8deb414396784555accb52179c1b03d2f2"} Oct 31 03:40:46.706326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:46.704420 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj"] Oct 31 03:40:46.706326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:46.704650 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" podUID="de472ad7-07c7-495d-8bc6-83ad091f4413" containerName="maas-api" containerID="cri-o://b3f81a945b339424aacd5173255b227c9e583c4f705eb5a81848d6a4917399bc" gracePeriod=30 Oct 31 03:40:47.049317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.043617 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:40:47.169819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.168063 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74235b76-3bed-49e7-98ae-a300f7be917f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74235b76-3bed-49e7-98ae-a300f7be917f-plugins-config-volume\") pod \"74235b76-3bed-49e7-98ae-a300f7be917f\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " Oct 31 03:40:47.169819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.168552 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74235b76-3bed-49e7-98ae-a300f7be917f-kube-api-access-4fgtp\" (UniqueName: \"kubernetes.io/projected/74235b76-3bed-49e7-98ae-a300f7be917f-kube-api-access-4fgtp\") pod \"74235b76-3bed-49e7-98ae-a300f7be917f\" (UID: \"74235b76-3bed-49e7-98ae-a300f7be917f\") " Oct 31 03:40:47.171247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.170984 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74235b76-3bed-49e7-98ae-a300f7be917f-plugins-config-volume" pod "74235b76-3bed-49e7-98ae-a300f7be917f" (UID: "74235b76-3bed-49e7-98ae-a300f7be917f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:47.176232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.173251 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt"] Oct 31 03:40:47.176232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.174836 5520 cpu_manager.go:414] "containerMap: removing container" podUID="74235b76-3bed-49e7-98ae-a300f7be917f" containerName="payload-processing" Oct 31 03:40:47.176232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.174856 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74235b76-3bed-49e7-98ae-a300f7be917f" containerName="payload-processing" Oct 31 03:40:47.176232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.175138 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="74235b76-3bed-49e7-98ae-a300f7be917f" containerName="payload-processing" Oct 31 03:40:47.235815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.210504 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.235815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.228140 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-73e222\"" Oct 31 03:40:47.280276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.274584 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-73e222-dockercfg-6x22b\"" Oct 31 03:40:47.280276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.276856 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh85b\" (UniqueName: \"kubernetes.io/projected/9101743e-f614-4906-bd37-af2d2e785874-kube-api-access-wh85b\") pod \"payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.280276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.277060 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9101743e-f614-4906-bd37-af2d2e785874-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.280276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.277534 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74235b76-3bed-49e7-98ae-a300f7be917f-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:47.321297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.315792 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt"] Oct 31 03:40:47.429227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.417381 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh85b\" (UniqueName: \"kubernetes.io/projected/9101743e-f614-4906-bd37-af2d2e785874-kube-api-access-wh85b\") pod \"payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.455274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.430009 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9101743e-f614-4906-bd37-af2d2e785874-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.468223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.431064 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9101743e-f614-4906-bd37-af2d2e785874-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.566385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.566347 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh85b\" (UniqueName: \"kubernetes.io/projected/9101743e-f614-4906-bd37-af2d2e785874-kube-api-access-wh85b\") pod \"payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.653442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.652242 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74235b76-3bed-49e7-98ae-a300f7be917f-kube-api-access-4fgtp" pod "74235b76-3bed-49e7-98ae-a300f7be917f" (UID: "74235b76-3bed-49e7-98ae-a300f7be917f"). InnerVolumeSpecName "kube-api-access-4fgtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:47.656247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.656219 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:47.662699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.662672 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgtp\" (UniqueName: \"kubernetes.io/projected/74235b76-3bed-49e7-98ae-a300f7be917f-kube-api-access-4fgtp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:47.738719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.738684 5520 generic.go:356] "Generic (PLEG): container finished" podID="de472ad7-07c7-495d-8bc6-83ad091f4413" containerID="b3f81a945b339424aacd5173255b227c9e583c4f705eb5a81848d6a4917399bc" exitCode=0 Oct 31 03:40:47.738903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.738779 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" event={"ID":"de472ad7-07c7-495d-8bc6-83ad091f4413","Type":"ContainerDied","Data":"b3f81a945b339424aacd5173255b227c9e583c4f705eb5a81848d6a4917399bc"} Oct 31 03:40:47.740705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.740635 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" event={"ID":"74235b76-3bed-49e7-98ae-a300f7be917f","Type":"ContainerDied","Data":"696b0f37f583cbbc6096f339939cbd885f5daf765da14fafda4fcbe830e03361"} Oct 31 03:40:47.740705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.740700 5520 scope.go:122] "RemoveContainer" containerID="386191c08999d1a33f76eab1bbe5ff8deb414396784555accb52179c1b03d2f2" Oct 31 03:40:47.740906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.740869 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb" Oct 31 03:40:47.822109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.822078 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb"] Oct 31 03:40:47.835928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.832537 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-74c760-78f4df97df-gpkgb"] Oct 31 03:40:47.874235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.873856 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:40:47.943626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.943589 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74235b76-3bed-49e7-98ae-a300f7be917f" path="/var/lib/kubelet/pods/74235b76-3bed-49e7-98ae-a300f7be917f/volumes" Oct 31 03:40:47.947000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.946976 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v"] Oct 31 03:40:47.948475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.948454 5520 cpu_manager.go:414] "containerMap: removing container" podUID="de472ad7-07c7-495d-8bc6-83ad091f4413" containerName="maas-api" Oct 31 03:40:47.948638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.948626 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de472ad7-07c7-495d-8bc6-83ad091f4413" containerName="maas-api" Oct 31 03:40:47.948948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.948935 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="de472ad7-07c7-495d-8bc6-83ad091f4413" containerName="maas-api" Oct 31 03:40:47.961145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.961117 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v"] Oct 31 03:40:47.961329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.961252 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:47.966836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.966813 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-007d82-serving-cert\"" Oct 31 03:40:47.975229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.972538 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de472ad7-07c7-495d-8bc6-83ad091f4413-kube-api-access-lnz6g\" (UniqueName: \"kubernetes.io/projected/de472ad7-07c7-495d-8bc6-83ad091f4413-kube-api-access-lnz6g\") pod \"de472ad7-07c7-495d-8bc6-83ad091f4413\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " Oct 31 03:40:47.975229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.972683 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-metrics-certs\") pod \"de472ad7-07c7-495d-8bc6-83ad091f4413\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " Oct 31 03:40:47.975229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.972779 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de472ad7-07c7-495d-8bc6-83ad091f4413-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de472ad7-07c7-495d-8bc6-83ad091f4413-openshift-service-ca\") pod \"de472ad7-07c7-495d-8bc6-83ad091f4413\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " Oct 31 03:40:47.975229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.972973 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-maas-api-tls\") pod \"de472ad7-07c7-495d-8bc6-83ad091f4413\" (UID: \"de472ad7-07c7-495d-8bc6-83ad091f4413\") " Oct 31 03:40:47.975229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.974861 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de472ad7-07c7-495d-8bc6-83ad091f4413-openshift-service-ca" pod "de472ad7-07c7-495d-8bc6-83ad091f4413" (UID: "de472ad7-07c7-495d-8bc6-83ad091f4413"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:47.982385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.980405 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-maas-api-tls" pod "de472ad7-07c7-495d-8bc6-83ad091f4413" (UID: "de472ad7-07c7-495d-8bc6-83ad091f4413"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:47.985428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.985402 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de472ad7-07c7-495d-8bc6-83ad091f4413-kube-api-access-lnz6g" pod "de472ad7-07c7-495d-8bc6-83ad091f4413" (UID: "de472ad7-07c7-495d-8bc6-83ad091f4413"). InnerVolumeSpecName "kube-api-access-lnz6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:47.987466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:47.987441 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-metrics-certs" pod "de472ad7-07c7-495d-8bc6-83ad091f4413" (UID: "de472ad7-07c7-495d-8bc6-83ad091f4413"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:40:48.065731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.065700 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt"] Oct 31 03:40:48.066178 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:40:48.066150 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9101743e_f614_4906_bd37_af2d2e785874.slice/crio-c0e596880cd8368f6550d7e77bd935a41e0b909899116dffbfb15784856561a8 WatchSource:0}: Error finding container c0e596880cd8368f6550d7e77bd935a41e0b909899116dffbfb15784856561a8: Status 404 returned error can't find the container with id c0e596880cd8368f6550d7e77bd935a41e0b909899116dffbfb15784856561a8 Oct 31 03:40:48.075148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075116 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-maas-api-tls\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.075496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075426 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-metrics-certs\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.075621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075600 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-openshift-service-ca\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.075687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075635 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8klc\" (UniqueName: \"kubernetes.io/projected/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-kube-api-access-t8klc\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.075789 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075776 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:48.075853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075796 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnz6g\" (UniqueName: \"kubernetes.io/projected/de472ad7-07c7-495d-8bc6-83ad091f4413-kube-api-access-lnz6g\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:48.075853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075812 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de472ad7-07c7-495d-8bc6-83ad091f4413-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:48.075946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.075853 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de472ad7-07c7-495d-8bc6-83ad091f4413-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:48.178106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.178069 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-maas-api-tls\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.178333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.178314 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-metrics-certs\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.178468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.178448 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-openshift-service-ca\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.178571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.178480 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8klc\" (UniqueName: \"kubernetes.io/projected/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-kube-api-access-t8klc\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.179306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.179272 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-openshift-service-ca\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.181352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.181325 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-maas-api-tls\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.182030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.182008 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-metrics-certs\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.190316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.190289 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8klc\" (UniqueName: \"kubernetes.io/projected/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-kube-api-access-t8klc\") pod \"maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.288806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.288702 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:48.588310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.588279 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v"] Oct 31 03:40:48.598089 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:40:48.597727 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3f6cb7_8536_4ada_bf93_5aa0c22b8dea.slice/crio-5c52ecb53aadbdf27554a162bf5778a26c5b7b09582c5219f6eefc39c51a9321 WatchSource:0}: Error finding container 5c52ecb53aadbdf27554a162bf5778a26c5b7b09582c5219f6eefc39c51a9321: Status 404 returned error can't find the container with id 5c52ecb53aadbdf27554a162bf5778a26c5b7b09582c5219f6eefc39c51a9321 Oct 31 03:40:48.600228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.600193 5520 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:40:48.793618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.793572 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" event={"ID":"9101743e-f614-4906-bd37-af2d2e785874","Type":"ContainerStarted","Data":"6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9"} Oct 31 03:40:48.793925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.793888 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" event={"ID":"9101743e-f614-4906-bd37-af2d2e785874","Type":"ContainerStarted","Data":"c0e596880cd8368f6550d7e77bd935a41e0b909899116dffbfb15784856561a8"} Oct 31 03:40:48.794557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.794529 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:40:48.796555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.796265 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" event={"ID":"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea","Type":"ContainerStarted","Data":"5c52ecb53aadbdf27554a162bf5778a26c5b7b09582c5219f6eefc39c51a9321"} Oct 31 03:40:48.797807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.797786 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" event={"ID":"de472ad7-07c7-495d-8bc6-83ad091f4413","Type":"ContainerDied","Data":"c3181b3c0972562f5dafebc85a6f77cd1745469ff2d9f4a61464c3577e10c056"} Oct 31 03:40:48.797911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.797840 5520 scope.go:122] "RemoveContainer" containerID="b3f81a945b339424aacd5173255b227c9e583c4f705eb5a81848d6a4917399bc" Oct 31 03:40:48.797911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.797795 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj" Oct 31 03:40:48.815167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.814947 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" podStartSLOduration=6.814929426 podStartE2EDuration="6.814929426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:48.813488753 +0000 UTC m=+1473.008011874" watchObservedRunningTime="2026-10-31 03:40:48.814929426 +0000 UTC m=+1473.009452548" Oct 31 03:40:48.837276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.835565 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj"] Oct 31 03:40:48.841446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:48.841416 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8361df-5b7996bfdd-nfrxj"] Oct 31 03:40:49.308704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.299532 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k"] Oct 31 03:40:49.338649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.338445 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k"] Oct 31 03:40:49.338649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.338561 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.354687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.354664 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b2dc82\"/\"default-dockercfg-x5mqd\"" Oct 31 03:40:49.354834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.354744 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b2dc82\"/\"test-model-b2dc82-kserve-self-signed-certs\"" Oct 31 03:40:49.365254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.363779 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b2dc82\"/\"kube-root-ca.crt\"" Oct 31 03:40:49.365254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.363888 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b2dc82\"/\"openshift-service-ca.crt\"" Oct 31 03:40:49.442422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.442390 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ddv\" (UniqueName: \"kubernetes.io/projected/d3a35073-afca-4267-aa1b-4c8fe008f6b5-kube-api-access-q2ddv\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.442877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.442856 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-home\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.443239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.443221 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-dshm\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.443748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.443730 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tmp-dir\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.443892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.443877 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-model-cache\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.444018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.444007 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tls-certs\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.551228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.548629 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ddv\" (UniqueName: \"kubernetes.io/projected/d3a35073-afca-4267-aa1b-4c8fe008f6b5-kube-api-access-q2ddv\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.551228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.548711 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-home\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.551228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.548928 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-dshm\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.551228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.548995 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tmp-dir\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.551228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.549029 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-model-cache\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.551228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.549105 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tls-certs\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.569228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.565778 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-home\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.569228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.566051 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tmp-dir\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.569228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.566342 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-model-cache\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.588224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.580881 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tls-certs\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.588224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.581147 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-dshm\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.617223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.607903 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ddv\" (UniqueName: \"kubernetes.io/projected/d3a35073-afca-4267-aa1b-4c8fe008f6b5-kube-api-access-q2ddv\") pod \"test-model-b2dc82-kserve-76d94b47c7-b2w5k\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.672228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.660470 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:49.869636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.862972 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" event={"ID":"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea","Type":"ContainerStarted","Data":"bbc501887b94151ee222b33f92c07a8066eb66b4482e3fa143abada52ebc8a4c"} Oct 31 03:40:49.869636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.864449 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:49.949294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:49.944643 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de472ad7-07c7-495d-8bc6-83ad091f4413" path="/var/lib/kubelet/pods/de472ad7-07c7-495d-8bc6-83ad091f4413/volumes" Oct 31 03:40:50.446857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:50.446773 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" podStartSLOduration=98.896679577 podStartE2EDuration="1m39.446754413s" totalImagesPullingTime="550.074836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:49.899678402 +0000 UTC m=+1474.094201523" watchObservedRunningTime="2026-10-31 03:40:50.446754413 +0000 UTC m=+1474.641277540" Oct 31 03:40:50.452737 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:40:50.452706 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3a35073_afca_4267_aa1b_4c8fe008f6b5.slice/crio-64a08152c101e2fc0cca833eb9f170c7ba4cd4efd8e7a72cd4934655d1d840f5 WatchSource:0}: Error finding container 64a08152c101e2fc0cca833eb9f170c7ba4cd4efd8e7a72cd4934655d1d840f5: Status 404 returned error can't find the container with id 64a08152c101e2fc0cca833eb9f170c7ba4cd4efd8e7a72cd4934655d1d840f5 Oct 31 03:40:50.454112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:50.454086 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k"] Oct 31 03:40:50.919287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:50.917057 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" event={"ID":"d3a35073-afca-4267-aa1b-4c8fe008f6b5","Type":"ContainerStarted","Data":"1519da2c89be388e2db8aab4993f471cac6af9c6133d7a6ab33b5a2c52d012b5"} Oct 31 03:40:50.919287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:50.917103 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" event={"ID":"d3a35073-afca-4267-aa1b-4c8fe008f6b5","Type":"ContainerStarted","Data":"64a08152c101e2fc0cca833eb9f170c7ba4cd4efd8e7a72cd4934655d1d840f5"} Oct 31 03:40:50.938094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:50.936305 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podStartSLOduration=1.936284471 podStartE2EDuration="1.936284471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:50.935625941 +0000 UTC m=+1475.130149088" watchObservedRunningTime="2026-10-31 03:40:50.936284471 +0000 UTC m=+1475.130807593" Oct 31 03:40:51.307821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:51.307780 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk"] Oct 31 03:40:51.308498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:51.308085 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" containerID="cri-o://89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c" gracePeriod=30 Oct 31 03:40:51.661389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:51.661309 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:56.995842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:56.991473 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:40:57.037228 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:57.036506 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16930518_1a71_4a09_9bd0_b139791c0169.slice/crio-89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16930518_1a71_4a09_9bd0_b139791c0169.slice/crio-conmon-89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:40:57.171250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.171192 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:40:57.263682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.263637 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-data\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.263866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.263764 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-socket\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.263956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.263914 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-data" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:57.264018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.263921 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-certs\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.264018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.263993 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-socket" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:57.264140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264086 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-kube-api-access-q6c7s\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-kube-api-access-q6c7s\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.264140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264118 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-envoy\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.264267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264145 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-certs" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:57.264267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264213 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-credential-socket\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.264267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264246 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-istio-token\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-istio-token\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.264493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264351 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16930518-1a71-4a09-9bd0-b139791c0169-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16930518-1a71-4a09-9bd0-b139791c0169-istio-podinfo\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.264493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264385 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16930518-1a71-4a09-9bd0-b139791c0169-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16930518-1a71-4a09-9bd0-b139791c0169-istiod-ca-cert\") pod \"16930518-1a71-4a09-9bd0-b139791c0169\" (UID: \"16930518-1a71-4a09-9bd0-b139791c0169\") " Oct 31 03:40:57.264602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264490 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-credential-socket" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:57.264790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.264766 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16930518-1a71-4a09-9bd0-b139791c0169-istiod-ca-cert" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:57.265369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.265342 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16930518-1a71-4a09-9bd0-b139791c0169-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.265369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.265371 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.265567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.265385 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.265567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.265400 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.265567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.265414 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.267119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.266955 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16930518-1a71-4a09-9bd0-b139791c0169-istio-podinfo" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:40:57.267119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.267093 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-istio-token" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:57.267321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.267171 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-envoy" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:57.269191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.269166 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-kube-api-access-q6c7s" pod "16930518-1a71-4a09-9bd0-b139791c0169" (UID: "16930518-1a71-4a09-9bd0-b139791c0169"). InnerVolumeSpecName "kube-api-access-q6c7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:57.270613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.270589 5520 generic.go:356] "Generic (PLEG): container finished" podID="16930518-1a71-4a09-9bd0-b139791c0169" containerID="89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c" exitCode=0 Oct 31 03:40:57.270818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.270804 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" event={"ID":"16930518-1a71-4a09-9bd0-b139791c0169","Type":"ContainerDied","Data":"89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c"} Oct 31 03:40:57.270891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.270843 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" event={"ID":"16930518-1a71-4a09-9bd0-b139791c0169","Type":"ContainerDied","Data":"6a05cda470f7589a7d4d786f4bc7950ce0f78e04db63e8547f3c534da879309d"} Oct 31 03:40:57.270891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.270863 5520 scope.go:122] "RemoveContainer" containerID="89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c" Oct 31 03:40:57.271009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.270976 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk" Oct 31 03:40:57.324091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.324065 5520 scope.go:122] "RemoveContainer" containerID="89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c" Oct 31 03:40:57.324570 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:40:57.324543 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c\": container with ID starting with 89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c not found: ID does not exist" containerID="89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c" Oct 31 03:40:57.324675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.324584 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c"} err="failed to get container status \"89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c\": rpc error: code = NotFound desc = could not find container \"89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c\": container with ID starting with 89f8c1c946efca2a5b02610db1a24aa54ebe870e522fb8774a63cd6fc308460c not found: ID does not exist" Oct 31 03:40:57.333361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.333326 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk"] Oct 31 03:40:57.337937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.337910 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-wr8wk"] Oct 31 03:40:57.367194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.367158 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6c7s\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-kube-api-access-q6c7s\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.367351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.367195 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16930518-1a71-4a09-9bd0-b139791c0169-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.367351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.367233 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16930518-1a71-4a09-9bd0-b139791c0169-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.367351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.367249 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16930518-1a71-4a09-9bd0-b139791c0169-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:57.924856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:57.924813 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16930518-1a71-4a09-9bd0-b139791c0169" path="/var/lib/kubelet/pods/16930518-1a71-4a09-9bd0-b139791c0169/volumes" Oct 31 03:40:59.661740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:59.661703 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:59.678334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:59.678301 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:40:59.869701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:40:59.869672 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:41:00.418103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:00.418075 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:41:06.794884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:06.794846 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch"] Oct 31 03:41:06.795478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:06.795195 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" podUID="95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" containerName="maas-api" containerID="cri-o://f0449984b5afdc459945161d7b3d281a0b8b1972d1966b12c8afc9be80a5543d" gracePeriod=30 Oct 31 03:41:07.353909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.350047 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w"] Oct 31 03:41:07.353909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.351649 5520 cpu_manager.go:414] "containerMap: removing container" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" Oct 31 03:41:07.353909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.351673 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" Oct 31 03:41:07.353909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.351882 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="16930518-1a71-4a09-9bd0-b139791c0169" containerName="istio-proxy" Oct 31 03:41:07.513314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.457817 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w"] Oct 31 03:41:07.513314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.459162 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.618190 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-880e04\"/\"kube-root-ca.crt\"" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.618916 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-880e04\"/\"default-dockercfg-vmvsn\"" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.619015 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-880e04\"/\"test-model-880e04-kserve-self-signed-certs\"" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.640011 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-880e04\"/\"openshift-service-ca.crt\"" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.647349 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-home\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.647464 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdm6b\" (UniqueName: \"kubernetes.io/projected/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-kube-api-access-mdm6b\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.647525 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-dshm\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.647679 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tmp-dir\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.647750 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tls-certs\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.678453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.647779 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-model-cache\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.902344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.892333 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdm6b\" (UniqueName: \"kubernetes.io/projected/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-kube-api-access-mdm6b\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.902344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.892432 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-dshm\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.902344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.892659 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tmp-dir\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.902344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.892748 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tls-certs\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.902344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.892773 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-model-cache\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.902344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.892882 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-home\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.909485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.908930 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-home\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.927194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.916820 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-model-cache\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:07.927194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:07.917139 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tmp-dir\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:08.191134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.169968 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tls-certs\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:08.191134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.170267 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-dshm\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:08.191134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.174022 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdm6b\" (UniqueName: \"kubernetes.io/projected/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-kube-api-access-mdm6b\") pod \"test-model-880e04-kserve-5ff89d8dbf-zfx4w\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:08.214930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.214896 5520 generic.go:356] "Generic (PLEG): container finished" podID="95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" containerID="f0449984b5afdc459945161d7b3d281a0b8b1972d1966b12c8afc9be80a5543d" exitCode=0 Oct 31 03:41:08.215309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.214946 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" event={"ID":"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55","Type":"ContainerDied","Data":"f0449984b5afdc459945161d7b3d281a0b8b1972d1966b12c8afc9be80a5543d"} Oct 31 03:41:08.231238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.231167 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:41:08.333641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.333561 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-kube-api-access-5ktgv\" (UniqueName: \"kubernetes.io/projected/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-kube-api-access-5ktgv\") pod \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " Oct 31 03:41:08.334305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.333708 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-openshift-service-ca\") pod \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " Oct 31 03:41:08.334305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.333803 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-metrics-certs\") pod \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " Oct 31 03:41:08.334305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.333872 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-maas-api-tls\") pod \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\" (UID: \"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55\") " Oct 31 03:41:08.334305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.334134 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-openshift-service-ca" pod "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" (UID: "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:08.334788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.334767 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:08.336664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.336633 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-maas-api-tls" pod "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" (UID: "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:08.338616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.338545 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-kube-api-access-5ktgv" pod "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" (UID: "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55"). InnerVolumeSpecName "kube-api-access-5ktgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:08.339838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.339812 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-metrics-certs" pod "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" (UID: "95cf461c-d9a4-4726-9c9a-f7c3b6efdd55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:08.434253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.434192 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:08.436317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.436290 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:08.436317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.436318 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktgv\" (UniqueName: \"kubernetes.io/projected/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-kube-api-access-5ktgv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:08.436493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.436337 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:08.806757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:08.803637 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w"] Oct 31 03:41:09.346781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:09.346730 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" event={"ID":"95cf461c-d9a4-4726-9c9a-f7c3b6efdd55","Type":"ContainerDied","Data":"3ec0ce440390b91e500954ad61eaf0f242983ff878cdc77fdf3b3c078e008658"} Oct 31 03:41:09.347329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:09.346802 5520 scope.go:122] "RemoveContainer" containerID="f0449984b5afdc459945161d7b3d281a0b8b1972d1966b12c8afc9be80a5543d" Oct 31 03:41:09.347329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:09.346973 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch" Oct 31 03:41:09.391004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:09.389951 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" event={"ID":"c04bd3bb-7308-4d36-95d1-d35a1d8b431b","Type":"ContainerStarted","Data":"afb9e6e6b06fab43ebe472d66b7cacb6ebb58c68081fc756e919a733075eea12"} Oct 31 03:41:09.424248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:09.421256 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch"] Oct 31 03:41:09.431908 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:41:09.431850 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:41:09.444689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:09.444655 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-74c760-86dbfb544b-tdtch"] Oct 31 03:41:09.921069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:09.921036 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" path="/var/lib/kubelet/pods/95cf461c-d9a4-4726-9c9a-f7c3b6efdd55/volumes" Oct 31 03:41:10.529228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:10.528704 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" event={"ID":"c04bd3bb-7308-4d36-95d1-d35a1d8b431b","Type":"ContainerStarted","Data":"383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217"} Oct 31 03:41:10.593231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:10.588134 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podStartSLOduration=3.588115868 podStartE2EDuration="3.588115868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:10.587762206 +0000 UTC m=+1494.782285328" watchObservedRunningTime="2026-10-31 03:41:10.588115868 +0000 UTC m=+1494.782638991" Oct 31 03:41:11.434570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:11.434536 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:18.446958 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:18.444279 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:18.516545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:18.516498 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:18.707714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:18.707633 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:29.201151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:29.201106 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr"] Oct 31 03:41:29.223602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:29.223572 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs"] Oct 31 03:41:34.202568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.202485 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" podUID="4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" containerName="payload-processing" containerID="cri-o://912f4fd67801256dfccae4bdcc9c4ad6e4055e05c69e3f0bbae94afd984fce00" gracePeriod=25 Oct 31 03:41:34.224173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.224114 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" podUID="f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" containerName="payload-pre-processing" containerID="cri-o://5aecb8a139708fc985f8caae54a3e7a3a53f986086f8882e15a20881e336568b" gracePeriod=25 Oct 31 03:41:34.652182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.652142 5520 generic.go:356] "Generic (PLEG): container finished" podID="f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" containerID="5aecb8a139708fc985f8caae54a3e7a3a53f986086f8882e15a20881e336568b" exitCode=0 Oct 31 03:41:34.652434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.652409 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" event={"ID":"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98","Type":"ContainerDied","Data":"5aecb8a139708fc985f8caae54a3e7a3a53f986086f8882e15a20881e336568b"} Oct 31 03:41:34.654923 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.654893 5520 generic.go:356] "Generic (PLEG): container finished" podID="4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" containerID="912f4fd67801256dfccae4bdcc9c4ad6e4055e05c69e3f0bbae94afd984fce00" exitCode=0 Oct 31 03:41:34.655011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.654953 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" event={"ID":"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49","Type":"ContainerDied","Data":"912f4fd67801256dfccae4bdcc9c4ad6e4055e05c69e3f0bbae94afd984fce00"} Oct 31 03:41:34.718241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.718194 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:41:34.747894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.747863 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:41:34.792711 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.792678 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-kube-api-access-kdj96\" (UniqueName: \"kubernetes.io/projected/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-kube-api-access-kdj96\") pod \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " Oct 31 03:41:34.792899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.792735 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-plugins-config-volume\") pod \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\" (UID: \"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98\") " Oct 31 03:41:34.792965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.792899 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-plugins-config-volume\") pod \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " Oct 31 03:41:34.793034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.793010 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-kube-api-access-5tjl4\" (UniqueName: \"kubernetes.io/projected/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-kube-api-access-5tjl4\") pod \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\" (UID: \"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49\") " Oct 31 03:41:34.793390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.793364 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-plugins-config-volume" pod "4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" (UID: "4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:34.793738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.793693 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:34.794118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.794098 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-plugins-config-volume" pod "f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" (UID: "f15df67d-2b23-4f0b-bc9a-4dda8cc9da98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:34.799107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.799076 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-kube-api-access-5tjl4" pod "4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" (UID: "4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49"). InnerVolumeSpecName "kube-api-access-5tjl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:34.803386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.803356 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-kube-api-access-kdj96" pod "f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" (UID: "f15df67d-2b23-4f0b-bc9a-4dda8cc9da98"). InnerVolumeSpecName "kube-api-access-kdj96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:34.895030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.894941 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tjl4\" (UniqueName: \"kubernetes.io/projected/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49-kube-api-access-5tjl4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:34.895030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.894970 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj96\" (UniqueName: \"kubernetes.io/projected/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-kube-api-access-kdj96\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:34.895030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:34.894980 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:35.665063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.664986 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" Oct 31 03:41:35.665063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.665017 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs" event={"ID":"f15df67d-2b23-4f0b-bc9a-4dda8cc9da98","Type":"ContainerDied","Data":"f29b1001112b5ae1ae90c2a735e782e4f2eef3ccf58a7b74dad667e33b71bb3e"} Oct 31 03:41:35.665563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.665079 5520 scope.go:122] "RemoveContainer" containerID="5aecb8a139708fc985f8caae54a3e7a3a53f986086f8882e15a20881e336568b" Oct 31 03:41:35.666658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.666634 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" Oct 31 03:41:35.666753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.666682 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr" event={"ID":"4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49","Type":"ContainerDied","Data":"4f845f47b1f83ee34fc3c969118ec04a2697bab9b0d86c82ae3e66792da99f2a"} Oct 31 03:41:35.680562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.680544 5520 scope.go:122] "RemoveContainer" containerID="912f4fd67801256dfccae4bdcc9c4ad6e4055e05c69e3f0bbae94afd984fce00" Oct 31 03:41:35.702451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.702425 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr"] Oct 31 03:41:35.713665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.713638 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3b986-5d7cff5867-c4lpr"] Oct 31 03:41:35.740669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.740638 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs"] Oct 31 03:41:35.744997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.744973 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3b986-66cdf65c47-v8rxs"] Oct 31 03:41:35.922732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.922658 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" path="/var/lib/kubelet/pods/4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49/volumes" Oct 31 03:41:35.923144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:35.923130 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" path="/var/lib/kubelet/pods/f15df67d-2b23-4f0b-bc9a-4dda8cc9da98/volumes" Oct 31 03:41:39.366669 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:41:39.366625 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:41:40.048409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.048372 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc"] Oct 31 03:41:40.050799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.050769 5520 cpu_manager.go:414] "containerMap: removing container" podUID="95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" containerName="maas-api" Oct 31 03:41:40.050988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.050975 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" containerName="maas-api" Oct 31 03:41:40.051104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.051093 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" containerName="payload-pre-processing" Oct 31 03:41:40.051190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.051178 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" containerName="payload-pre-processing" Oct 31 03:41:40.051386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.051373 5520 cpu_manager.go:414] "containerMap: removing container" podUID="4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" containerName="payload-processing" Oct 31 03:41:40.051475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.051464 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" containerName="payload-processing" Oct 31 03:41:40.052012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.051992 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea0f260-91a1-4ec3-ba35-8edfcb6a2b49" containerName="payload-processing" Oct 31 03:41:40.052157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.052147 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cf461c-d9a4-4726-9c9a-f7c3b6efdd55" containerName="maas-api" Oct 31 03:41:40.052271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.052260 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15df67d-2b23-4f0b-bc9a-4dda8cc9da98" containerName="payload-pre-processing" Oct 31 03:41:40.062328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.062295 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.083678 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5dcb7\"/\"kube-root-ca.crt\"" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.084175 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5dcb7\"/\"default-dockercfg-kmpnt\"" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.084260 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5dcb7\"/\"openshift-service-ca.crt\"" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.084427 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5dcb7\"/\"test-model-c5dcb7-kserve-self-signed-certs\"" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.135730 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qk65\" (UniqueName: \"kubernetes.io/projected/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-kube-api-access-2qk65\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.135802 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-dshm\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.135835 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-home\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.135879 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tls-certs\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.135919 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tmp-dir\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.140316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.136224 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-model-cache\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.196225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.181270 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc"] Oct 31 03:41:40.266280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.252768 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qk65\" (UniqueName: \"kubernetes.io/projected/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-kube-api-access-2qk65\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.266280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.252844 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-dshm\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.266280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.252886 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-home\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.266280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.252925 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tls-certs\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.266280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.252955 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tmp-dir\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.266280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.253316 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-model-cache\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.266280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.254020 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-model-cache\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.354278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.339786 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-home\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.354278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.340076 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tmp-dir\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.518272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.502156 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qk65\" (UniqueName: \"kubernetes.io/projected/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-kube-api-access-2qk65\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.697961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.688628 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tls-certs\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.797227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.784551 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-dshm\") pod \"test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:40.888226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:40.884565 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:41.102414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:41.102192 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:41.554092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:41.525986 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:42.117096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.097131 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": context deadline exceeded" Oct 31 03:41:42.217019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.216977 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:41:42.311242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.310660 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:42.331644 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:41:42.331611 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded3bfa65_ac7b_4b46_81d3_8319d1f54937.slice/crio-f5a151ccc82467929b5868ea38dd9f5239726106aeead6642db2930f9e95ff8f WatchSource:0}: Error finding container f5a151ccc82467929b5868ea38dd9f5239726106aeead6642db2930f9e95ff8f: Status 404 returned error can't find the container with id f5a151ccc82467929b5868ea38dd9f5239726106aeead6642db2930f9e95ff8f Oct 31 03:41:42.417862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.409388 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc"] Oct 31 03:41:42.419834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.419802 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:42.559306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.557324 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:42.575391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.560296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:41:42.575391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.572273 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:41:42.575391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.573951 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:42.611695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.590861 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:42.615488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.615408 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:42.816229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.808997 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:41:42.844234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:42.831303 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:43.067227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.066791 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:43.097533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.097485 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:43.139604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.139560 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:41:43.169274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.168487 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:43.169274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.169018 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:41:43.321281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.312447 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:41:43.321281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.314758 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:41:43.494325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.487509 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" event={"ID":"ed3bfa65-ac7b-4b46-81d3-8319d1f54937","Type":"ContainerStarted","Data":"f5a151ccc82467929b5868ea38dd9f5239726106aeead6642db2930f9e95ff8f"} Oct 31 03:41:43.819226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.802828 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:41:43.852228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.833326 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:43.852228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:43.834037 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:44.067291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.067176 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.078114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.078091 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:41:44.124963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.124910 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.127605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.127576 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:41:44.223060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.191266 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": context deadline exceeded" Oct 31 03:41:44.245917 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.245872 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.333384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.333264 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.351708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.351669 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.460231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.456868 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:44.526175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.526110 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.658469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.658336 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.833170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.833121 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:44.921288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.917456 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:41:44.938233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.922822 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:41:44.942263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.938556 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:41:44.947713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:44.947675 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" event={"ID":"ed3bfa65-ac7b-4b46-81d3-8319d1f54937","Type":"ContainerStarted","Data":"526c0e425966e82aea7da7b69143273257290df6a68cac9a2bec8f8077d8ec78"} Oct 31 03:41:45.094569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:45.094488 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podStartSLOduration=5.094448511 podStartE2EDuration="5.094448511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:45.04284037 +0000 UTC m=+1529.237363514" watchObservedRunningTime="2026-10-31 03:41:45.094448511 +0000 UTC m=+1529.288971634" Oct 31 03:41:45.895269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:45.895187 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:50.903575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:50.892057 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:50.940616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:50.940553 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:51.155025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:51.154946 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:52.271420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.271322 5520 generic.go:356] "Generic (PLEG): container finished" podID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerID="a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7" exitCode=0 Oct 31 03:41:52.272283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.272254 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" event={"ID":"929fc3a7-eaf6-4d34-8754-dec95c4aff02","Type":"ContainerDied","Data":"a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7"} Oct 31 03:41:52.275317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.275286 5520 generic.go:356] "Generic (PLEG): container finished" podID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerID="4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b" exitCode=0 Oct 31 03:41:52.275450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.275368 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" event={"ID":"82d3f882-054f-4e82-b27e-1a0c6cfda587","Type":"ContainerDied","Data":"4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b"} Oct 31 03:41:52.276157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.276124 5520 scope.go:122] "RemoveContainer" containerID="4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b" Oct 31 03:41:52.276921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.276896 5520 scope.go:122] "RemoveContainer" containerID="a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7" Oct 31 03:41:52.279250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.279184 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:41:52.279354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.279286 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:41:52.279354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.279340 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:41:52.280715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.280680 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:41:52.280829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.280764 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:41:52.280829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:52.280817 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:41:53.293257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.293172 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" event={"ID":"929fc3a7-eaf6-4d34-8754-dec95c4aff02","Type":"ContainerStarted","Data":"d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870"} Oct 31 03:41:53.312459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.312431 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" event={"ID":"82d3f882-054f-4e82-b27e-1a0c6cfda587","Type":"ContainerStarted","Data":"78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c"} Oct 31 03:41:53.446347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.446318 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:41:53.447193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.447165 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:41:53.447275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.447240 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:41:53.677286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.674534 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:41:53.686507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.684321 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:41:53.686507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:53.684378 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:41:54.446952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:54.446916 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:41:54.447433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:54.446973 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:41:54.529442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:54.529380 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:41:54.672982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:54.671842 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:41:54.672982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:54.671900 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:41:55.122228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.121252 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq"] Oct 31 03:41:55.122228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.121522 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" podUID="62e55949-1870-4a62-8a9a-cc22317fb556" containerName="maas-api" containerID="cri-o://5d969d565dc1a0a0a48dd5d8bddc489ea4a5cf0751bdf78c4ccce7bcd88a0589" gracePeriod=30 Oct 31 03:41:55.318427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.315359 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:41:55.458467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.457986 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:41:55.458467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.458041 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:41:55.499231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.487075 5520 generic.go:356] "Generic (PLEG): container finished" podID="62e55949-1870-4a62-8a9a-cc22317fb556" containerID="5d969d565dc1a0a0a48dd5d8bddc489ea4a5cf0751bdf78c4ccce7bcd88a0589" exitCode=0 Oct 31 03:41:55.499231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.488517 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" event={"ID":"62e55949-1870-4a62-8a9a-cc22317fb556","Type":"ContainerDied","Data":"5d969d565dc1a0a0a48dd5d8bddc489ea4a5cf0751bdf78c4ccce7bcd88a0589"} Oct 31 03:41:55.671025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.670985 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:41:55.671864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:55.671829 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:41:56.238486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.202116 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:56.261856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.261814 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:56.318767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.318706 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:56.457993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.457953 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:41:56.458349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.458316 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:56.459266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.458909 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:41:56.459266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.459182 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:56.534412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.532291 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:56.539281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.536708 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:41:56.539281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.536771 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:41:56.683163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.683124 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:41:56.684335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.684309 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:41:56.684569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.684531 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:41:56.793930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.793807 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e55949-1870-4a62-8a9a-cc22317fb556-kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/62e55949-1870-4a62-8a9a-cc22317fb556-kube-api-access-5b5fr\") pod \"62e55949-1870-4a62-8a9a-cc22317fb556\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " Oct 31 03:41:56.794121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.794010 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e55949-1870-4a62-8a9a-cc22317fb556-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e55949-1870-4a62-8a9a-cc22317fb556-openshift-service-ca\") pod \"62e55949-1870-4a62-8a9a-cc22317fb556\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " Oct 31 03:41:56.802225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.794293 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-maas-api-tls\") pod \"62e55949-1870-4a62-8a9a-cc22317fb556\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " Oct 31 03:41:56.820844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.820809 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e55949-1870-4a62-8a9a-cc22317fb556-openshift-service-ca" pod "62e55949-1870-4a62-8a9a-cc22317fb556" (UID: "62e55949-1870-4a62-8a9a-cc22317fb556"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:56.859454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.855980 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:56.859699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.859677 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-metrics-certs\") pod \"62e55949-1870-4a62-8a9a-cc22317fb556\" (UID: \"62e55949-1870-4a62-8a9a-cc22317fb556\") " Oct 31 03:41:56.886273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.860813 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e55949-1870-4a62-8a9a-cc22317fb556-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:56.921786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.921750 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-maas-api-tls" pod "62e55949-1870-4a62-8a9a-cc22317fb556" (UID: "62e55949-1870-4a62-8a9a-cc22317fb556"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:56.947355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.947259 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-metrics-certs" pod "62e55949-1870-4a62-8a9a-cc22317fb556" (UID: "62e55949-1870-4a62-8a9a-cc22317fb556"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:56.948134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:56.948090 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e55949-1870-4a62-8a9a-cc22317fb556-kube-api-access-5b5fr" pod "62e55949-1870-4a62-8a9a-cc22317fb556" (UID: "62e55949-1870-4a62-8a9a-cc22317fb556"). InnerVolumeSpecName "kube-api-access-5b5fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:57.024234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.015332 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:57.024234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.015369 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e55949-1870-4a62-8a9a-cc22317fb556-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:57.024234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.015385 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b5fr\" (UniqueName: \"kubernetes.io/projected/62e55949-1870-4a62-8a9a-cc22317fb556-kube-api-access-5b5fr\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:57.609250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.603800 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:41:57.609250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.603864 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:41:57.711227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.701356 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:57.803229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.794100 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:41:57.803229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.794173 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:41:57.930987 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.927386 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:57.931813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:57.931788 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" Oct 31 03:41:58.126231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.121694 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.162305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.161263 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq"] Oct 31 03:41:58.162305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.161318 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq" event={"ID":"62e55949-1870-4a62-8a9a-cc22317fb556","Type":"ContainerDied","Data":"78f2d8bd41442ba0cd0eef7b4657cd844ee12359340b0e3ffc74d52e99a63eaa"} Oct 31 03:41:58.162305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.161352 5520 scope.go:122] "RemoveContainer" containerID="5d969d565dc1a0a0a48dd5d8bddc489ea4a5cf0751bdf78c4ccce7bcd88a0589" Oct 31 03:41:58.220224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.214319 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.231255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.229388 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.259320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.256458 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-34dbeb-644767c54c-nn7hq"] Oct 31 03:41:58.296268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.296228 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.393218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.393156 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.444679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.444632 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:41:58.465541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.465457 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:41:58.468981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.468948 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:41:58.484058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.470650 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.532855 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:41:58.532815 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e55949_1870_4a62_8a9a_cc22317fb556.slice\": RecentStats: unable to find data in memory cache]" Oct 31 03:41:58.610301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.609930 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.683919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.683881 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:41:58.684458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.684427 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:41:58.827283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.810328 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.829289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.820497 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:58.851230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:58.841521 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.142543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.142422 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:41:59.227972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.227929 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Oct 31 03:41:59.228967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.228913 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:41:59.231415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.229977 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.231933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.231917 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:41:59.300950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.300301 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.300950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.300938 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:41:59.424281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.421467 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.460246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.456869 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:41:59.460246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.456940 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:41:59.460246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.458305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:41:59.460246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.459861 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:41:59.477826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.462309 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.477826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.462686 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:41:59.818190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.818148 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.836523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.819334 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.886261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.876362 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:41:59.956957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.888239 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.968947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.888272 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:41:59.994869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.947346 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:41:59.994869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.947399 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:41:59.994869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:41:59.956879 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:42:00.087277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.086493 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.087277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.086726 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.152602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.148586 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.160648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.158367 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.294237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.289992 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": context deadline exceeded" Oct 31 03:42:00.294237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.291423 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.306436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.306402 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e55949-1870-4a62-8a9a-cc22317fb556" path="/var/lib/kubelet/pods/62e55949-1870-4a62-8a9a-cc22317fb556/volumes" Oct 31 03:42:00.315918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.315884 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:42:00.317491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.317451 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.317503 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb"] Oct 31 03:42:00.344315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.341735 5520 cpu_manager.go:414] "containerMap: removing container" podUID="62e55949-1870-4a62-8a9a-cc22317fb556" containerName="maas-api" Oct 31 03:42:00.344315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.341760 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e55949-1870-4a62-8a9a-cc22317fb556" containerName="maas-api" Oct 31 03:42:00.344315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.342133 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e55949-1870-4a62-8a9a-cc22317fb556" containerName="maas-api" Oct 31 03:42:00.408989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.408963 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb"] Oct 31 03:42:00.409315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.409300 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.412424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.411659 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.414349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.413552 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-73e222\"/\"kube-root-ca.crt\"" Oct 31 03:42:00.414349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.413651 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-73e222\"/\"default-dockercfg-kd9ww\"" Oct 31 03:42:00.414349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.414220 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-73e222\"/\"test-model-73e222-kserve-self-signed-certs\"" Oct 31 03:42:00.414349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.414300 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-73e222\"/\"openshift-service-ca.crt\"" Oct 31 03:42:00.434016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.433977 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.446762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.446720 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:42:00.468623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.459969 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:42:00.468623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.462296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:42:00.468623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.463779 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.537304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.537272 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0694d1ee-0802-47de-a550-53f1da6e183b-tls-certs\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.556228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.537633 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-model-cache\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.556696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.556667 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-home\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.592861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.592812 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-dshm\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.593183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.593148 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-tmp-dir\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.593487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.593470 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgrv\" (UniqueName: \"kubernetes.io/projected/0694d1ee-0802-47de-a550-53f1da6e183b-kube-api-access-4jgrv\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.622028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.559032 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:42:00.622028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.621629 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:42:00.622028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.620098 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.671833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.671702 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:42:00.671833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.671756 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:42:00.702724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.679400 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.726224 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0694d1ee-0802-47de-a550-53f1da6e183b-tls-certs\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.726311 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-model-cache\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.726480 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-home\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.726506 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-dshm\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.726528 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-tmp-dir\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.726624 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgrv\" (UniqueName: \"kubernetes.io/projected/0694d1ee-0802-47de-a550-53f1da6e183b-kube-api-access-4jgrv\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.729719 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-home\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.734255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.729967 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-model-cache\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.752471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.744717 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-tmp-dir\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.752471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.752050 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-dshm\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.844628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.831241 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgrv\" (UniqueName: \"kubernetes.io/projected/0694d1ee-0802-47de-a550-53f1da6e183b-kube-api-access-4jgrv\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.987274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.956283 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0694d1ee-0802-47de-a550-53f1da6e183b-tls-certs\") pod \"test-model-73e222-kserve-59bd9c5445-fldxb\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:00.987274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.978857 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:42:00.996701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.996626 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:00.997904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:00.997881 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:42:01.071248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.054336 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.071248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.054776 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:42:01.100824 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.100775 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:42:01.101345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.101320 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:01.102095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.101362 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:42:01.205391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.203185 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.205391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.203513 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.205391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.204042 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:42:01.205391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.204102 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:42:01.247420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.247327 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:42:01.335249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.309418 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": context deadline exceeded" Oct 31 03:42:01.335249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.311373 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.335249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.322306 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.419390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.413724 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.419390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.414632 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:42:01.446263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.446224 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:42:01.447487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.447457 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:42:01.451309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.451276 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:42:01.451498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.447701 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:42:01.466347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.466294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:42:01.466673 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.466307 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:42:01.565351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.565250 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.623223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.604581 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:42:01.673239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.665079 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:42:01.723231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.697477 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:42:01.723231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.697533 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:42:01.807383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.805642 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:42:01.865429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.865151 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:01.914237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.913113 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:42:01.971943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:01.971267 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:02.007188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.005129 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:02.071262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.071223 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:02.120192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.120108 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:42:02.160060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.160019 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:42:02.173300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.173274 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:42:02.173590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.173577 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:42:02.251358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.251332 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:42:02.252328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.252299 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:42:02.339796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.339746 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:42:02.368786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.354528 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:42:02.457628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.457470 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:42:02.563800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.563764 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:42:02.698228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.690230 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 03:42:02.721406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:02.721313 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 03:42:03.083319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.083285 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt"] Oct 31 03:42:03.314222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.313773 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:42:03.314222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.313818 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt"] Oct 31 03:42:03.315426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.314545 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.362704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.362098 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-86a40d-dockercfg-qbgt9\"" Oct 31 03:42:03.362704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.362661 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-86a40d\"" Oct 31 03:42:03.393616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.389498 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb"] Oct 31 03:42:03.472367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.460993 5520 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 03:42:03.472367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.461049 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 03:42:03.497274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.497237 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:42:03.497883 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.497855 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:42:03.534391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.531425 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:03.557839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.556773 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7mx\" (UniqueName: \"kubernetes.io/projected/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-kube-api-access-8h7mx\") pod \"payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.557839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.556854 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.691253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.687955 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7mx\" (UniqueName: \"kubernetes.io/projected/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-kube-api-access-8h7mx\") pod \"payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.691253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.688059 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.691253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.689755 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.744498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.732524 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7mx\" (UniqueName: \"kubernetes.io/projected/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-kube-api-access-8h7mx\") pod \"payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.857827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.857555 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:03.892722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.892687 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:42:03.906249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:03.904174 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:42:04.256928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.256893 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt"] Oct 31 03:42:04.269321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.269283 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" event={"ID":"0694d1ee-0802-47de-a550-53f1da6e183b","Type":"ContainerStarted","Data":"7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89"} Oct 31 03:42:04.269453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.269325 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" event={"ID":"0694d1ee-0802-47de-a550-53f1da6e183b","Type":"ContainerStarted","Data":"d01606cdf0f2796817c10a460ae63b7dac95f93d947db4ea1c9a77fa7afff9b8"} Oct 31 03:42:04.287773 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.287642 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podStartSLOduration=5.287623621 podStartE2EDuration="5.287623621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:04.287483744 +0000 UTC m=+1548.482006866" watchObservedRunningTime="2026-10-31 03:42:04.287623621 +0000 UTC m=+1548.482146742" Oct 31 03:42:04.532943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.527676 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:42:04.547278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.538142 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:42:04.655815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.655783 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz"] Oct 31 03:42:04.656400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:04.656349 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" containerID="cri-o://a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51" gracePeriod=30 Oct 31 03:42:05.111231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:05.108574 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:05.280148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:05.280098 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" event={"ID":"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6","Type":"ContainerStarted","Data":"ff5f99dd04f716ea631432f027e0f47347cbe1cd1d6e0cae09ce79622afa0c8a"} Oct 31 03:42:05.280965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:05.280171 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" event={"ID":"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6","Type":"ContainerStarted","Data":"49a8309e8aa4731624aba55dda40184c242ca73b9ad83cc02b6703e609675460"} Oct 31 03:42:05.280965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:05.280271 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:05.303462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:05.303411 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" podStartSLOduration=3.303395929 podStartE2EDuration="3.303395929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:05.298696567 +0000 UTC m=+1549.493219690" watchObservedRunningTime="2026-10-31 03:42:05.303395929 +0000 UTC m=+1549.497919081" Oct 31 03:42:09.361591 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:42:09.361548 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:42:10.006937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.006898 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:42:10.192255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192220 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istiod-ca-cert\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192296 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-socket\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192363 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-kube-api-access-xbvlr\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-kube-api-access-xbvlr\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192400 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-credential-socket\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192513 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-certs\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192547 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-podinfo\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192579 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-data\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192604 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-socket" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:42:10.192785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192675 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istiod-ca-cert" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:42:10.192785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192687 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-credential-socket" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:42:10.192785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192709 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-envoy\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.192785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192771 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-certs" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:42:10.193020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192778 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-token\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-token\") pod \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\" (UID: \"44a96e65-fb7f-4bc9-88f7-2bf66a97d460\") " Oct 31 03:42:10.193020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.192901 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-data" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:42:10.194023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.193993 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.194160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.194028 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.194160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.194045 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.194160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.194061 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.194160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.194075 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.195346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.195320 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-token" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:42:10.195427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.195396 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-envoy" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:42:10.195427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.195411 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-podinfo" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:42:10.197325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.197303 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-kube-api-access-xbvlr" pod "44a96e65-fb7f-4bc9-88f7-2bf66a97d460" (UID: "44a96e65-fb7f-4bc9-88f7-2bf66a97d460"). InnerVolumeSpecName "kube-api-access-xbvlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:42:10.295676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.295640 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbvlr\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-kube-api-access-xbvlr\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.295676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.295669 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.295676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.295680 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.295676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.295688 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a96e65-fb7f-4bc9-88f7-2bf66a97d460-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:10.351604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.351565 5520 generic.go:356] "Generic (PLEG): container finished" podID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerID="a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51" exitCode=0 Oct 31 03:42:10.351812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.351644 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" Oct 31 03:42:10.351812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.351641 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" event={"ID":"44a96e65-fb7f-4bc9-88f7-2bf66a97d460","Type":"ContainerDied","Data":"a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51"} Oct 31 03:42:10.351812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.351804 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz" event={"ID":"44a96e65-fb7f-4bc9-88f7-2bf66a97d460","Type":"ContainerDied","Data":"7be273c7080a3e7cf2558cb76c0da6059a7ee4864d571aa7047e91b4ac59d1ea"} Oct 31 03:42:10.352010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.351837 5520 scope.go:122] "RemoveContainer" containerID="a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51" Oct 31 03:42:10.366765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.366746 5520 scope.go:122] "RemoveContainer" containerID="482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90" Oct 31 03:42:10.408077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.404627 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz"] Oct 31 03:42:10.408077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.406368 5520 scope.go:122] "RemoveContainer" containerID="a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51" Oct 31 03:42:10.413921 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:42:10.412436 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51\": container with ID starting with a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51 not found: ID does not exist" containerID="a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51" Oct 31 03:42:10.413921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.412496 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51"} err="failed to get container status \"a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51\": rpc error: code = NotFound desc = could not find container \"a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51\": container with ID starting with a24407942c621c15408aa782ffc830d42c0a32412bf1de0fe7586f468103fb51 not found: ID does not exist" Oct 31 03:42:10.413921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.412540 5520 scope.go:122] "RemoveContainer" containerID="482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90" Oct 31 03:42:10.413921 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:42:10.412999 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90\": container with ID starting with 482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90 not found: ID does not exist" containerID="482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90" Oct 31 03:42:10.413921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.413028 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90"} err="failed to get container status \"482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90\": rpc error: code = NotFound desc = could not find container \"482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90\": container with ID starting with 482cdb1b4afcffc3930c0f94e16cd9081dc5d847e8c4eb52a48c030c0580bd90 not found: ID does not exist" Oct 31 03:42:10.454233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.448269 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-34dbeb-openshift-default-75bcf5645c-ngktz"] Oct 31 03:42:10.812448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.811970 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp"] Oct 31 03:42:10.812448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.812271 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" podUID="9cf77425-e97a-48e6-8d80-751e612ee153" containerName="maas-api" containerID="cri-o://d3e945e4f2a1860a7fb1d7b143e5b585d3ba692cae2c63d3de5fa91548a76b3b" gracePeriod=30 Oct 31 03:42:10.845228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:10.844529 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" podUID="9cf77425-e97a-48e6-8d80-751e612ee153" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.163:8443/health\": dial tcp 10.134.0.163:8443: connect: connection refused" Oct 31 03:42:11.106282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.105120 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:11.785999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.784543 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:11.788969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.787307 5520 generic.go:356] "Generic (PLEG): container finished" podID="9cf77425-e97a-48e6-8d80-751e612ee153" containerID="d3e945e4f2a1860a7fb1d7b143e5b585d3ba692cae2c63d3de5fa91548a76b3b" exitCode=0 Oct 31 03:42:11.788969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.787591 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" event={"ID":"9cf77425-e97a-48e6-8d80-751e612ee153","Type":"ContainerDied","Data":"d3e945e4f2a1860a7fb1d7b143e5b585d3ba692cae2c63d3de5fa91548a76b3b"} Oct 31 03:42:11.834626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.834596 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:42:11.888515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.888467 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:42:11.919820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.919784 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-maas-api-tls\") pod \"9cf77425-e97a-48e6-8d80-751e612ee153\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " Oct 31 03:42:11.920007 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.919840 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf77425-e97a-48e6-8d80-751e612ee153-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf77425-e97a-48e6-8d80-751e612ee153-openshift-service-ca\") pod \"9cf77425-e97a-48e6-8d80-751e612ee153\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " Oct 31 03:42:11.920932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.920554 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-metrics-certs\") pod \"9cf77425-e97a-48e6-8d80-751e612ee153\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " Oct 31 03:42:11.921189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.921170 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf77425-e97a-48e6-8d80-751e612ee153-kube-api-access-w72m9\" (UniqueName: \"kubernetes.io/projected/9cf77425-e97a-48e6-8d80-751e612ee153-kube-api-access-w72m9\") pod \"9cf77425-e97a-48e6-8d80-751e612ee153\" (UID: \"9cf77425-e97a-48e6-8d80-751e612ee153\") " Oct 31 03:42:11.922897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.920825 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf77425-e97a-48e6-8d80-751e612ee153-openshift-service-ca" pod "9cf77425-e97a-48e6-8d80-751e612ee153" (UID: "9cf77425-e97a-48e6-8d80-751e612ee153"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:42:11.924005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.923984 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf77425-e97a-48e6-8d80-751e612ee153-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:11.924322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.924244 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-maas-api-tls" pod "9cf77425-e97a-48e6-8d80-751e612ee153" (UID: "9cf77425-e97a-48e6-8d80-751e612ee153"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:42:11.931301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.930130 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf77425-e97a-48e6-8d80-751e612ee153-kube-api-access-w72m9" pod "9cf77425-e97a-48e6-8d80-751e612ee153" (UID: "9cf77425-e97a-48e6-8d80-751e612ee153"). InnerVolumeSpecName "kube-api-access-w72m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:42:11.932829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.932804 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-metrics-certs" pod "9cf77425-e97a-48e6-8d80-751e612ee153" (UID: "9cf77425-e97a-48e6-8d80-751e612ee153"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:42:11.937522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:11.937493 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" path="/var/lib/kubelet/pods/44a96e65-fb7f-4bc9-88f7-2bf66a97d460/volumes" Oct 31 03:42:12.041500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.041361 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:12.041500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.041397 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf77425-e97a-48e6-8d80-751e612ee153-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:12.041500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.041414 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72m9\" (UniqueName: \"kubernetes.io/projected/9cf77425-e97a-48e6-8d80-751e612ee153-kube-api-access-w72m9\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:12.826153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.826118 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" Oct 31 03:42:12.826153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.826142 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp" event={"ID":"9cf77425-e97a-48e6-8d80-751e612ee153","Type":"ContainerDied","Data":"068c4bc395cb82dad5f9d97c4fe066ab3461e2be6ed8e617fb2d15f2e8d39013"} Oct 31 03:42:12.827135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.826258 5520 scope.go:122] "RemoveContainer" containerID="d3e945e4f2a1860a7fb1d7b143e5b585d3ba692cae2c63d3de5fa91548a76b3b" Oct 31 03:42:12.850645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.850583 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp"] Oct 31 03:42:12.854933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:12.854902 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3b986-84bbb97fc7-8bpdp"] Oct 31 03:42:13.922072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:13.922040 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf77425-e97a-48e6-8d80-751e612ee153" path="/var/lib/kubelet/pods/9cf77425-e97a-48e6-8d80-751e612ee153/volumes" Oct 31 03:42:15.493560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:15.493524 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc"] Oct 31 03:42:15.518327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:15.518294 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf"] Oct 31 03:42:16.292407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:16.292376 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:42:17.907526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.907472 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq"] Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.914761 5520 cpu_manager.go:414] "containerMap: removing container" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.914789 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.914812 5520 cpu_manager.go:414] "containerMap: removing container" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.914820 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.914895 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9cf77425-e97a-48e6-8d80-751e612ee153" containerName="maas-api" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.914902 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf77425-e97a-48e6-8d80-751e612ee153" containerName="maas-api" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.915145 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.915160 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a96e65-fb7f-4bc9-88f7-2bf66a97d460" containerName="istio-proxy" Oct 31 03:42:17.922663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.915179 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf77425-e97a-48e6-8d80-751e612ee153" containerName="maas-api" Oct 31 03:42:17.929745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.929702 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.937417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.937240 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-46ebfe-openshift-default-dockercfg-zlk2v\"" Oct 31 03:42:17.955013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.954979 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.957822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.957798 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.959556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.958098 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.959787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.959764 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw59b\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-kube-api-access-rw59b\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.961017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.960996 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.961375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.961334 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.961881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.961546 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.961881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.961590 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.961881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.961617 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:17.963397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:17.963374 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq"] Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.077978 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078124 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078180 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078231 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078256 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078428 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078487 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078578 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.078679 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw59b\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-kube-api-access-rw59b\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.120239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.079548 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.155141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.155107 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.155657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.155633 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.156064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.156043 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.192276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.163675 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.192556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.176772 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.204226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.196487 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.243227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.231601 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.258323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.244386 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw59b\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-kube-api-access-rw59b\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.271780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.270591 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:18.837243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.837180 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq"] Oct 31 03:42:18.838639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.838606 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:42:18.838874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.838821 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:42:18.839121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:18.839086 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:42:19.020281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:19.020149 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" event={"ID":"7e0d8f04-2f35-4581-a012-9e205b8ced0e","Type":"ContainerStarted","Data":"7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a"} Oct 31 03:42:19.020941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:19.020912 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" event={"ID":"7e0d8f04-2f35-4581-a012-9e205b8ced0e","Type":"ContainerStarted","Data":"aac06d68a57a51a13861fef1542a0fe726b897edeb4f58bc9b9f78a44e5e85be"} Oct 31 03:42:19.052286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:19.052225 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podStartSLOduration=2.052185545 podStartE2EDuration="2.052185545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:19.047649461 +0000 UTC m=+1563.242172584" watchObservedRunningTime="2026-10-31 03:42:19.052185545 +0000 UTC m=+1563.246708666" Oct 31 03:42:19.293970 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:19.293891 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:19.338319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:19.313035 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:42:19.338319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:19.313090 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:42:20.278103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.278070 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:20.494513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.494455 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" podUID="10446300-c01e-4947-bcc2-cb29c9a36345" containerName="payload-processing" containerID="cri-o://95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce" gracePeriod=25 Oct 31 03:42:20.519278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.519224 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" podUID="60e0ce96-2b1f-44b2-a657-f7eed6e62911" containerName="payload-pre-processing" containerID="cri-o://1114c161b3dffab14ca3c055c67b52a60f90909882b5d9aec2e7cc24fb4b629c" gracePeriod=25 Oct 31 03:42:20.542835 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:20.542775 5520 logging.go:55] [core] [Channel #3570 SubChannel #3571]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.151:50052", ServerName: "10.134.0.151:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.151:50052: connect: connection refused" Oct 31 03:42:20.542835 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:20.542781 5520 logging.go:55] [core] [Channel #3569 SubChannel #3572]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.150:50052", ServerName: "10.134.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.150:50052: connect: connection refused" Oct 31 03:42:20.732454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.731880 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx"] Oct 31 03:42:20.744272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.743569 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx"] Oct 31 03:42:20.744272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.743692 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:20.764510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.759512 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50b5f2-openshift-default-dockercfg-pxfcz\"" Oct 31 03:42:20.971034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.968800 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-data\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:20.968972 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-credential-socket\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.116884 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbtx\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-kube-api-access-4dbtx\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.116939 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-envoy\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.116972 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-certs\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.117173 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-socket\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.117221 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-podinfo\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.117249 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-token\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.117784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.117318 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istiod-ca-cert\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.306504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.290405 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:42:21.316023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.315970 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-credential-socket\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.351363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.349422 5520 generic.go:356] "Generic (PLEG): container finished" podID="60e0ce96-2b1f-44b2-a657-f7eed6e62911" containerID="1114c161b3dffab14ca3c055c67b52a60f90909882b5d9aec2e7cc24fb4b629c" exitCode=0 Oct 31 03:42:21.351694 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.349453 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" event={"ID":"60e0ce96-2b1f-44b2-a657-f7eed6e62911","Type":"ContainerDied","Data":"1114c161b3dffab14ca3c055c67b52a60f90909882b5d9aec2e7cc24fb4b629c"} Oct 31 03:42:21.372395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.351118 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-credential-socket\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.372683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.353278 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:42:21.381386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.370465 5520 generic.go:356] "Generic (PLEG): container finished" podID="10446300-c01e-4947-bcc2-cb29c9a36345" containerID="95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce" exitCode=0 Oct 31 03:42:21.382221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.370709 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" Oct 31 03:42:21.382536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.372604 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" event={"ID":"10446300-c01e-4947-bcc2-cb29c9a36345","Type":"ContainerDied","Data":"95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce"} Oct 31 03:42:21.382806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.382770 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" event={"ID":"10446300-c01e-4947-bcc2-cb29c9a36345","Type":"ContainerDied","Data":"0132ebf31da7b8f598c5b60c554bdff022b12b560772513374f1c199aafbe164"} Oct 31 03:42:21.394808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.383034 5520 scope.go:122] "RemoveContainer" containerID="95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.401527 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.401679 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.403029 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbtx\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-kube-api-access-4dbtx\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.403120 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-envoy\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.404076 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-certs\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.404507 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-certs\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.404754 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-socket\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.404804 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-podinfo\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.406343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.405616 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-socket\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.430457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.405672 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-token\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.431421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.431390 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istiod-ca-cert\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.432239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.432136 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-data\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.432737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.432711 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-data\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.480001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.479966 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istiod-ca-cert\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.547240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.543759 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" podUID="60e0ce96-2b1f-44b2-a657-f7eed6e62911" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.151:50052\" within 1s: context deadline exceeded" Oct 31 03:42:21.547240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.544132 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc" podUID="10446300-c01e-4947-bcc2-cb29c9a36345" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.150:50052\" within 1s: context deadline exceeded" Oct 31 03:42:21.547240 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:21.544285 5520 logging.go:55] [core] [Channel #3569 SubChannel #3572]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.150:50052", ServerName: "10.134.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.150:50052: operation was canceled" Oct 31 03:42:21.612220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.609328 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-podinfo\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.612220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.609875 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbtx\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-kube-api-access-4dbtx\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.612220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.609923 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10446300-c01e-4947-bcc2-cb29c9a36345-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10446300-c01e-4947-bcc2-cb29c9a36345-plugins-config-volume\") pod \"10446300-c01e-4947-bcc2-cb29c9a36345\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " Oct 31 03:42:21.612220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.610162 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e0ce96-2b1f-44b2-a657-f7eed6e62911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e0ce96-2b1f-44b2-a657-f7eed6e62911-plugins-config-volume\") pod \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " Oct 31 03:42:21.612220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.610290 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10446300-c01e-4947-bcc2-cb29c9a36345-kube-api-access-7hmgx\" (UniqueName: \"kubernetes.io/projected/10446300-c01e-4947-bcc2-cb29c9a36345-kube-api-access-7hmgx\") pod \"10446300-c01e-4947-bcc2-cb29c9a36345\" (UID: \"10446300-c01e-4947-bcc2-cb29c9a36345\") " Oct 31 03:42:21.612220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.610460 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e0ce96-2b1f-44b2-a657-f7eed6e62911-kube-api-access-kpssz\" (UniqueName: \"kubernetes.io/projected/60e0ce96-2b1f-44b2-a657-f7eed6e62911-kube-api-access-kpssz\") pod \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\" (UID: \"60e0ce96-2b1f-44b2-a657-f7eed6e62911\") " Oct 31 03:42:21.616073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.616049 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e0ce96-2b1f-44b2-a657-f7eed6e62911-plugins-config-volume" pod "60e0ce96-2b1f-44b2-a657-f7eed6e62911" (UID: "60e0ce96-2b1f-44b2-a657-f7eed6e62911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:42:21.622478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.622397 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e0ce96-2b1f-44b2-a657-f7eed6e62911-kube-api-access-kpssz" pod "60e0ce96-2b1f-44b2-a657-f7eed6e62911" (UID: "60e0ce96-2b1f-44b2-a657-f7eed6e62911"). InnerVolumeSpecName "kube-api-access-kpssz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:42:21.625110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.625068 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10446300-c01e-4947-bcc2-cb29c9a36345-kube-api-access-7hmgx" pod "10446300-c01e-4947-bcc2-cb29c9a36345" (UID: "10446300-c01e-4947-bcc2-cb29c9a36345"). InnerVolumeSpecName "kube-api-access-7hmgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:42:21.627149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.625571 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10446300-c01e-4947-bcc2-cb29c9a36345-plugins-config-volume" pod "10446300-c01e-4947-bcc2-cb29c9a36345" (UID: "10446300-c01e-4947-bcc2-cb29c9a36345"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:42:21.629388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.628761 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-envoy\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.631286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.630997 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-token\") pod \"e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.635363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.635180 5520 scope.go:122] "RemoveContainer" containerID="95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce" Oct 31 03:42:21.635615 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:42:21.635596 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce\": container with ID starting with 95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce not found: ID does not exist" containerID="95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce" Oct 31 03:42:21.635687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.635622 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce"} err="failed to get container status \"95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce\": rpc error: code = NotFound desc = could not find container \"95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce\": container with ID starting with 95eabb5ac74269c612641b96d657860d894a3ca02a4797427b71d1a5d2a33cce not found: ID does not exist" Oct 31 03:42:21.680890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.680549 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:21.753649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.744772 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e0ce96-2b1f-44b2-a657-f7eed6e62911-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:21.753649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.744807 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmgx\" (UniqueName: \"kubernetes.io/projected/10446300-c01e-4947-bcc2-cb29c9a36345-kube-api-access-7hmgx\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:21.753649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.744821 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpssz\" (UniqueName: \"kubernetes.io/projected/60e0ce96-2b1f-44b2-a657-f7eed6e62911-kube-api-access-kpssz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:21.753649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:21.744835 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10446300-c01e-4947-bcc2-cb29c9a36345-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:42:22.059239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.051835 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc"] Oct 31 03:42:22.073997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.073948 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-sdvbc"] Oct 31 03:42:22.339691 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:22.339662 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5efbc60_6ff8_43eb_bfa4_62687f95d1bf.slice/crio-95e3c5801597f6b8080bbb925e028383001873418f17cdff38be9c1eebe7939b WatchSource:0}: Error finding container 95e3c5801597f6b8080bbb925e028383001873418f17cdff38be9c1eebe7939b: Status 404 returned error can't find the container with id 95e3c5801597f6b8080bbb925e028383001873418f17cdff38be9c1eebe7939b Oct 31 03:42:22.341461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.341438 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx"] Oct 31 03:42:22.343492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.343460 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:42:22.343559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.343525 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:42:22.343559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.343552 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:42:22.397091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.397060 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" Oct 31 03:42:22.397297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.397065 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf" event={"ID":"60e0ce96-2b1f-44b2-a657-f7eed6e62911","Type":"ContainerDied","Data":"e043eb8cca4a40027877540fae5138873784a43034fc4650d4c66c66f3ff6127"} Oct 31 03:42:22.397297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.397243 5520 scope.go:122] "RemoveContainer" containerID="1114c161b3dffab14ca3c055c67b52a60f90909882b5d9aec2e7cc24fb4b629c" Oct 31 03:42:22.403136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.403111 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" event={"ID":"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf","Type":"ContainerStarted","Data":"95e3c5801597f6b8080bbb925e028383001873418f17cdff38be9c1eebe7939b"} Oct 31 03:42:22.436338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.436306 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf"] Oct 31 03:42:22.456361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.446449 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-x59hf"] Oct 31 03:42:22.958270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.958234 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5"] Oct 31 03:42:22.959615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.959587 5520 cpu_manager.go:414] "containerMap: removing container" podUID="10446300-c01e-4947-bcc2-cb29c9a36345" containerName="payload-processing" Oct 31 03:42:22.959615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.959616 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10446300-c01e-4947-bcc2-cb29c9a36345" containerName="payload-processing" Oct 31 03:42:22.959779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.959697 5520 cpu_manager.go:414] "containerMap: removing container" podUID="60e0ce96-2b1f-44b2-a657-f7eed6e62911" containerName="payload-pre-processing" Oct 31 03:42:22.959779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.959705 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e0ce96-2b1f-44b2-a657-f7eed6e62911" containerName="payload-pre-processing" Oct 31 03:42:22.959899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.959886 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e0ce96-2b1f-44b2-a657-f7eed6e62911" containerName="payload-pre-processing" Oct 31 03:42:22.959940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.959916 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="10446300-c01e-4947-bcc2-cb29c9a36345" containerName="payload-processing" Oct 31 03:42:22.964900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.964877 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:22.983952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.983927 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-46ebfe-dockercfg-jhzwp\"" Oct 31 03:42:22.991230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:22.990815 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5"] Oct 31 03:42:23.011083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.010589 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-46ebfe\"" Oct 31 03:42:23.041526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.027159 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9wf\" (UniqueName: \"kubernetes.io/projected/24611e04-1978-4f47-b8d3-4234b0ce003e-kube-api-access-qn9wf\") pod \"payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:23.041526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.027317 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24611e04-1978-4f47-b8d3-4234b0ce003e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:23.135230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.131766 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24611e04-1978-4f47-b8d3-4234b0ce003e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:23.159448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.139715 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9wf\" (UniqueName: \"kubernetes.io/projected/24611e04-1978-4f47-b8d3-4234b0ce003e-kube-api-access-qn9wf\") pod \"payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:23.195450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.195410 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24611e04-1978-4f47-b8d3-4234b0ce003e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:23.336718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.335064 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9wf\" (UniqueName: \"kubernetes.io/projected/24611e04-1978-4f47-b8d3-4234b0ce003e-kube-api-access-qn9wf\") pod \"payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:23.375586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.357299 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:23.570483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.570451 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" event={"ID":"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf","Type":"ContainerStarted","Data":"30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c"} Oct 31 03:42:23.617275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.613081 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podStartSLOduration=3.613061623 podStartE2EDuration="3.613061623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:23.599315625 +0000 UTC m=+1567.793838747" watchObservedRunningTime="2026-10-31 03:42:23.613061623 +0000 UTC m=+1567.807584745" Oct 31 03:42:23.692582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.692556 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:23.715574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.707525 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:24.010938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.990589 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10446300-c01e-4947-bcc2-cb29c9a36345" path="/var/lib/kubelet/pods/10446300-c01e-4947-bcc2-cb29c9a36345/volumes" Oct 31 03:42:24.010938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.992953 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e0ce96-2b1f-44b2-a657-f7eed6e62911" path="/var/lib/kubelet/pods/60e0ce96-2b1f-44b2-a657-f7eed6e62911/volumes" Oct 31 03:42:24.010938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:23.994714 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p"] Oct 31 03:42:24.089624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.089588 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p"] Oct 31 03:42:24.095473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.090754 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.115221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.105430 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-86a40d-serving-cert\"" Oct 31 03:42:24.115221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.105504 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5"] Oct 31 03:42:24.154219 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:24.153518 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24611e04_1978_4f47_b8d3_4234b0ce003e.slice/crio-b0dcdd5988fb602355aa5695050326d7262b62069554079264f75cb981f8c5ae WatchSource:0}: Error finding container b0dcdd5988fb602355aa5695050326d7262b62069554079264f75cb981f8c5ae: Status 404 returned error can't find the container with id b0dcdd5988fb602355aa5695050326d7262b62069554079264f75cb981f8c5ae Oct 31 03:42:24.192185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.192149 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa02636b-059a-434e-b5a0-5d93f8313dc3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.192356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.192229 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-maas-api-tls\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.192356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.192259 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks7cd\" (UniqueName: \"kubernetes.io/projected/aa02636b-059a-434e-b5a0-5d93f8313dc3-kube-api-access-ks7cd\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.192356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.192341 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-metrics-certs\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.294295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.294247 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa02636b-059a-434e-b5a0-5d93f8313dc3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.294925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.294887 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-maas-api-tls\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.295083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.295062 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks7cd\" (UniqueName: \"kubernetes.io/projected/aa02636b-059a-434e-b5a0-5d93f8313dc3-kube-api-access-ks7cd\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.295315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.295300 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-metrics-certs\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.299076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.299036 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa02636b-059a-434e-b5a0-5d93f8313dc3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.302304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.302255 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-maas-api-tls\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.304741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.304719 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-metrics-certs\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.308555 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.308514 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks7cd\" (UniqueName: \"kubernetes.io/projected/aa02636b-059a-434e-b5a0-5d93f8313dc3-kube-api-access-ks7cd\") pod \"maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.504333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.503777 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:24.734280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.728016 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" event={"ID":"24611e04-1978-4f47-b8d3-4234b0ce003e","Type":"ContainerStarted","Data":"fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7"} Oct 31 03:42:24.734280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.728097 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:24.734280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.728124 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:24.734280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.728144 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" event={"ID":"24611e04-1978-4f47-b8d3-4234b0ce003e","Type":"ContainerStarted","Data":"b0dcdd5988fb602355aa5695050326d7262b62069554079264f75cb981f8c5ae"} Oct 31 03:42:24.734280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.729477 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:42:24.808054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:24.800593 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" podStartSLOduration=2.8005719940000002 podStartE2EDuration="2.800571994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:24.776946205 +0000 UTC m=+1568.971469353" watchObservedRunningTime="2026-10-31 03:42:24.800571994 +0000 UTC m=+1568.995095116" Oct 31 03:42:25.374460 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:25.374426 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa02636b_059a_434e_b5a0_5d93f8313dc3.slice/crio-0f52ec69a8b664a9ff8083df23996f23845f1cef6685898bf2fe373c2dbbee28 WatchSource:0}: Error finding container 0f52ec69a8b664a9ff8083df23996f23845f1cef6685898bf2fe373c2dbbee28: Status 404 returned error can't find the container with id 0f52ec69a8b664a9ff8083df23996f23845f1cef6685898bf2fe373c2dbbee28 Oct 31 03:42:25.374898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:25.374435 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p"] Oct 31 03:42:25.736294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:25.736252 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" event={"ID":"aa02636b-059a-434e-b5a0-5d93f8313dc3","Type":"ContainerStarted","Data":"0f52ec69a8b664a9ff8083df23996f23845f1cef6685898bf2fe373c2dbbee28"} Oct 31 03:42:26.747449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:26.747400 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" event={"ID":"aa02636b-059a-434e-b5a0-5d93f8313dc3","Type":"ContainerStarted","Data":"8646ee0c6a2302dbb7a4df8d7a9ff6deb0e01ae0ed8bc63c899b63c54a96d119"} Oct 31 03:42:26.747978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:26.747510 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:26.773084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:26.772438 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" podStartSLOduration=3.365604057 podStartE2EDuration="3.772421357s" totalImagesPullingTime="406.8173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:26.771883432 +0000 UTC m=+1570.966406555" watchObservedRunningTime="2026-10-31 03:42:26.772421357 +0000 UTC m=+1570.966944479" Oct 31 03:42:32.790196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:32.790162 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:42:35.746375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:35.746332 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:42:39.167040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.165308 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.167040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.165457 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.284675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.284630 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.357224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.322717 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.404227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.394537 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.445343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.445299 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.445880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.445851 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:42:39.447855 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.447829 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.448703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.448679 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.449096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.448824 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.467763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.467722 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.594594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.594547 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.680343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.680294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.709282 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:42:39.703443 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:42:39.735651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.735606 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:39.736257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:39.736228 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 31 03:42:40.032282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:40.031363 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.450288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:40.449846 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:40.532318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:40.513928 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:42:40.559820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:40.554901 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:42:42.814956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.814921 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45"] Oct 31 03:42:42.821339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.821313 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:42.840285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.840256 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-46ebfe-serving-cert\"" Oct 31 03:42:42.846316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.846191 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45"] Oct 31 03:42:42.917380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.917346 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:42.917670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.917644 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:42.917793 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.917775 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-metrics-certs\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:42.917863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:42.917832 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtdj\" (UniqueName: \"kubernetes.io/projected/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-kube-api-access-grtdj\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.057733 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.057690 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.058067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.058044 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-metrics-certs\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.058234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.058196 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtdj\" (UniqueName: \"kubernetes.io/projected/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-kube-api-access-grtdj\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.058503 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.058485 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.074147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.059691 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.087225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.083295 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-metrics-certs\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.148722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.148681 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.168067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.168031 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtdj\" (UniqueName: \"kubernetes.io/projected/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-kube-api-access-grtdj\") pod \"maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:43.466301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:43.464987 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:44.121044 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:44.121010 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbde46c5_a7f8_4d6e_9452_ccda66a974e4.slice/crio-a4dc975ccfc58fc5c65daebb04bab4994b0418bfe4714ab57ac8a5586e13092f WatchSource:0}: Error finding container a4dc975ccfc58fc5c65daebb04bab4994b0418bfe4714ab57ac8a5586e13092f: Status 404 returned error can't find the container with id a4dc975ccfc58fc5c65daebb04bab4994b0418bfe4714ab57ac8a5586e13092f Oct 31 03:42:44.124662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:44.124634 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45"] Oct 31 03:42:44.525513 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:44.525474 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" event={"ID":"dbde46c5-a7f8-4d6e-9452-ccda66a974e4","Type":"ContainerStarted","Data":"a4dc975ccfc58fc5c65daebb04bab4994b0418bfe4714ab57ac8a5586e13092f"} Oct 31 03:42:45.537159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:45.537127 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" event={"ID":"dbde46c5-a7f8-4d6e-9452-ccda66a974e4","Type":"ContainerStarted","Data":"84bd4308c2950e33de37632e236598623ffa41628385fc7290f20950d5df9307"} Oct 31 03:42:45.537629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:45.537363 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:45.558586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:45.558534 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" podStartSLOduration=3.102794019 podStartE2EDuration="3.558517539s" totalImagesPullingTime="455.72352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:45.553313975 +0000 UTC m=+1589.747837106" watchObservedRunningTime="2026-10-31 03:42:45.558517539 +0000 UTC m=+1589.753040661" Oct 31 03:42:47.196196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.196156 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh"] Oct 31 03:42:47.202526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.202500 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.218582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.216016 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-50b5f2-serving-cert\"" Oct 31 03:42:47.219384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.219357 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh"] Oct 31 03:42:47.280578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.280543 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315c9d44-5d61-47a5-8bf7-85172dea8af2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.280788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.280607 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48jh\" (UniqueName: \"kubernetes.io/projected/315c9d44-5d61-47a5-8bf7-85172dea8af2-kube-api-access-x48jh\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.281034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.281009 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-metrics-certs\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.281191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.281174 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.383145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.383113 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-metrics-certs\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.383352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.383268 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.383352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.383341 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315c9d44-5d61-47a5-8bf7-85172dea8af2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.383488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.383393 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48jh\" (UniqueName: \"kubernetes.io/projected/315c9d44-5d61-47a5-8bf7-85172dea8af2-kube-api-access-x48jh\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.384869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.384751 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315c9d44-5d61-47a5-8bf7-85172dea8af2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.390282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.386988 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.425225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.419911 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-metrics-certs\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.427986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.427954 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48jh\" (UniqueName: \"kubernetes.io/projected/315c9d44-5d61-47a5-8bf7-85172dea8af2-kube-api-access-x48jh\") pod \"maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.538355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.533506 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:47.848560 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:47.848528 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315c9d44_5d61_47a5_8bf7_85172dea8af2.slice/crio-c5bebca95dbc7a3a28b671591e488014408f1fbf1f26db38dd0e01eb6ca050ba WatchSource:0}: Error finding container c5bebca95dbc7a3a28b671591e488014408f1fbf1f26db38dd0e01eb6ca050ba: Status 404 returned error can't find the container with id c5bebca95dbc7a3a28b671591e488014408f1fbf1f26db38dd0e01eb6ca050ba Oct 31 03:42:47.849265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:47.849157 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh"] Oct 31 03:42:48.585333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:48.585293 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" event={"ID":"315c9d44-5d61-47a5-8bf7-85172dea8af2","Type":"ContainerStarted","Data":"10fb61703c5ba4849b167ea977b23997d6e1c8d35369e2ad2109033b91bcd3f1"} Oct 31 03:42:48.585823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:48.585360 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" event={"ID":"315c9d44-5d61-47a5-8bf7-85172dea8af2","Type":"ContainerStarted","Data":"c5bebca95dbc7a3a28b671591e488014408f1fbf1f26db38dd0e01eb6ca050ba"} Oct 31 03:42:48.585823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:48.585576 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:48.607410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:48.607362 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" podStartSLOduration=1.069372849 podStartE2EDuration="1.607347208s" totalImagesPullingTime="537.974359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:48.602016953 +0000 UTC m=+1592.796540075" watchObservedRunningTime="2026-10-31 03:42:48.607347208 +0000 UTC m=+1592.801870328" Oct 31 03:42:51.551487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:51.551460 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:42:54.414370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.414320 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6"] Oct 31 03:42:54.433045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.433006 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6"] Oct 31 03:42:54.433258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.433144 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.439620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.439592 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5d9b6\"/\"kube-root-ca.crt\"" Oct 31 03:42:54.440601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.440575 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5d9b6\"/\"test-model-a5d9b6-kserve-self-signed-certs\"" Oct 31 03:42:54.440719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.440577 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a5d9b6\"/\"openshift-service-ca.crt\"" Oct 31 03:42:54.440765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.440719 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a5d9b6\"/\"default-dockercfg-mt56m\"" Oct 31 03:42:54.492612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.492380 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-home\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.492612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.492428 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tmp-dir\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.492612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.492519 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-dshm\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.492612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.492566 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmf2\" (UniqueName: \"kubernetes.io/projected/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-kube-api-access-wlmf2\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.492991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.492802 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-model-cache\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.492991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.492946 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tls-certs\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.597229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.597166 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-model-cache\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.597449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.597351 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tls-certs\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.597518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.597470 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-home\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.597518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.597495 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tmp-dir\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.597631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.597568 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-dshm\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.597631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.597611 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmf2\" (UniqueName: \"kubernetes.io/projected/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-kube-api-access-wlmf2\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.601232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.597792 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-model-cache\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.601232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.598407 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tmp-dir\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.601232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.598671 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-home\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.786567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.776636 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:42:54.828235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.826595 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-dshm\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:54.857230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:54.854886 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmf2\" (UniqueName: \"kubernetes.io/projected/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-kube-api-access-wlmf2\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:55.042269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:55.041829 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tls-certs\") pod \"test-model-a5d9b6-kserve-79b884f49-v4lq6\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:55.090026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:55.089984 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:42:55.402276 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:42:55.402181 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eddc40_82f6_4177_92d2_6bbcfcca1c2a.slice/crio-9c5224b1c0d71a4cfd278cb4e15e866c0baabdf01e35051238097c1d89493692 WatchSource:0}: Error finding container 9c5224b1c0d71a4cfd278cb4e15e866c0baabdf01e35051238097c1d89493692: Status 404 returned error can't find the container with id 9c5224b1c0d71a4cfd278cb4e15e866c0baabdf01e35051238097c1d89493692 Oct 31 03:42:55.403508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:55.403477 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6"] Oct 31 03:42:55.799878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:55.799837 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" event={"ID":"23eddc40-82f6-4177-92d2-6bbcfcca1c2a","Type":"ContainerStarted","Data":"1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80"} Oct 31 03:42:55.800430 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:55.799891 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" event={"ID":"23eddc40-82f6-4177-92d2-6bbcfcca1c2a","Type":"ContainerStarted","Data":"9c5224b1c0d71a4cfd278cb4e15e866c0baabdf01e35051238097c1d89493692"} Oct 31 03:42:55.823295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:55.823238 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podStartSLOduration=1.823223779 podStartE2EDuration="1.823223779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:55.819807342 +0000 UTC m=+1600.014330487" watchObservedRunningTime="2026-10-31 03:42:55.823223779 +0000 UTC m=+1600.017746897" Oct 31 03:42:56.091495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:42:56.091408 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:43:05.091218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:05.091173 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:43:05.106841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:05.106816 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:43:05.898750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:05.898722 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:43:09.360271 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:43:09.360231 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:43:15.373942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:15.373889 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v"] Oct 31 03:43:15.374968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:15.374937 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" podUID="3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" containerName="maas-api" containerID="cri-o://bbc501887b94151ee222b33f92c07a8066eb66b4482e3fa143abada52ebc8a4c" gracePeriod=30 Oct 31 03:43:16.258912 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.258878 5520 generic.go:356] "Generic (PLEG): container finished" podID="3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" containerID="bbc501887b94151ee222b33f92c07a8066eb66b4482e3fa143abada52ebc8a4c" exitCode=0 Oct 31 03:43:16.259546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.259515 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" event={"ID":"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea","Type":"ContainerDied","Data":"bbc501887b94151ee222b33f92c07a8066eb66b4482e3fa143abada52ebc8a4c"} Oct 31 03:43:16.453860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.453828 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:43:16.494818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.494783 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x"] Oct 31 03:43:16.496243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.496170 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" containerName="maas-api" Oct 31 03:43:16.496243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.496196 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" containerName="maas-api" Oct 31 03:43:16.496453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.496444 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" containerName="maas-api" Oct 31 03:43:16.513562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.513492 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.516638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.516585 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x"] Oct 31 03:43:16.542102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.542052 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-openshift-service-ca\") pod \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " Oct 31 03:43:16.542341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.542120 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-kube-api-access-t8klc\" (UniqueName: \"kubernetes.io/projected/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-kube-api-access-t8klc\") pod \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " Oct 31 03:43:16.542341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.542157 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-maas-api-tls\") pod \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " Oct 31 03:43:16.542618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.542382 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-metrics-certs\") pod \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\" (UID: \"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea\") " Oct 31 03:43:16.543024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.542991 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543142 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543284 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543332 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543378 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87478e66-340f-438d-9ab0-0d7784eaa03d-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543479 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543644 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543623 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543663 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.543772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.543741 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zz5\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-kube-api-access-f2zz5\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.544291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.544257 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-openshift-service-ca" pod "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" (UID: "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:16.547380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.547358 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-maas-api-tls" pod "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" (UID: "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:43:16.550998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.550962 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-kube-api-access-t8klc" pod "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" (UID: "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea"). InnerVolumeSpecName "kube-api-access-t8klc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:16.551169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.551139 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-metrics-certs" pod "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" (UID: "3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:43:16.645491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.645381 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646032 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646082 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646221 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87478e66-340f-438d-9ab0-0d7784eaa03d-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646317 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646409 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646434 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646501 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zz5\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-kube-api-access-f2zz5\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646629 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646690 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646709 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.646797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646729 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:16.646797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646775 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8klc\" (UniqueName: \"kubernetes.io/projected/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-kube-api-access-t8klc\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:16.646989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.646961 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.647263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.647240 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.647585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.647535 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:16.647585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.647562 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:16.647890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.647863 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87478e66-340f-438d-9ab0-0d7784eaa03d-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.648740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.648700 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.649779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.649758 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.657991 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.657965 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zz5\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-kube-api-access-f2zz5\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.673616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.673592 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-h459x\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:16.838492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:16.838405 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:17.153665 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:43:17.153637 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87478e66_340f_438d_9ab0_0d7784eaa03d.slice/crio-1eb62c49c2917345a54330e30cf08a42ecb5e563f7c0b990d258667379d3ffea WatchSource:0}: Error finding container 1eb62c49c2917345a54330e30cf08a42ecb5e563f7c0b990d258667379d3ffea: Status 404 returned error can't find the container with id 1eb62c49c2917345a54330e30cf08a42ecb5e563f7c0b990d258667379d3ffea Oct 31 03:43:17.154161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.154137 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x"] Oct 31 03:43:17.157028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.156994 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:17.157110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.157063 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:17.157110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.157092 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:17.274240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.271388 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" event={"ID":"3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea","Type":"ContainerDied","Data":"5c52ecb53aadbdf27554a162bf5778a26c5b7b09582c5219f6eefc39c51a9321"} Oct 31 03:43:17.274240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.271468 5520 scope.go:122] "RemoveContainer" containerID="bbc501887b94151ee222b33f92c07a8066eb66b4482e3fa143abada52ebc8a4c" Oct 31 03:43:17.274240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.271643 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v" Oct 31 03:43:17.281327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.281285 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" event={"ID":"87478e66-340f-438d-9ab0-0d7784eaa03d","Type":"ContainerStarted","Data":"7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c"} Oct 31 03:43:17.281588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.281543 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" event={"ID":"87478e66-340f-438d-9ab0-0d7784eaa03d","Type":"ContainerStarted","Data":"1eb62c49c2917345a54330e30cf08a42ecb5e563f7c0b990d258667379d3ffea"} Oct 31 03:43:17.321060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.321006 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podStartSLOduration=13.320990313 podStartE2EDuration="13.320990313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:17.315650975 +0000 UTC m=+1621.510174097" watchObservedRunningTime="2026-10-31 03:43:17.320990313 +0000 UTC m=+1621.515513433" Oct 31 03:43:17.389954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.389860 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v"] Oct 31 03:43:17.416575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.416539 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-007d82-f6767b8dc-n2r7v"] Oct 31 03:43:17.838933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.838899 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:17.924892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:17.924855 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea" path="/var/lib/kubelet/pods/3a3f6cb7-8536-4ada-bf93-5aa0c22b8dea/volumes" Oct 31 03:43:18.853629 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:18.853594 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:43:18.854136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:18.853646 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:43:19.839777 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:19.839747 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:43:19.839931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:19.839801 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:43:20.853226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:20.841633 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:43:20.853226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:20.841698 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:43:21.839443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:21.839404 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:43:21.839632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:21.839460 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:43:22.839272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:22.839238 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:43:22.839732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:22.839294 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:43:23.839397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:23.839365 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 31 03:43:23.839886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:23.839422 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 31 03:43:24.887165 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:24.887133 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:24.901323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:24.901289 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:24.909301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:24.906692 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:43:35.858248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:35.836031 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7"] Oct 31 03:43:35.858248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:35.836318 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" containerID="cri-o://b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24" gracePeriod=30 Oct 31 03:43:36.468340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:36.468294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:43:36.469883 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:36.469846 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:43:36.800631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:36.800539 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:36.840227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:36.839817 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:36.959234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:36.924854 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:36.959234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:36.949854 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:36.981252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:36.980288 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.077894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.077812 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.146351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.146305 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.457228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.456364 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.477388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.473229 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:43:37.477388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.473965 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:43:37.508224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.503534 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.508224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.507812 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.508224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.508137 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:43:37.871286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.833388 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.871286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.834259 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:43:37.871286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.835568 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:43:37.915232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.907814 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:43:37.980238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.978624 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:37.993255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.989627 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.000334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:37.997981 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.015656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.015610 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.083227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.078787 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:43:38.097604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.097579 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:43:38.132425 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.132330 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:43:38.237428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.237385 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.260089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.260046 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.408284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.407742 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp"] Oct 31 03:43:38.408467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.408282 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.489645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.489234 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:43:38.510079 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.510034 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.621454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.621421 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp"] Oct 31 03:43:38.621883 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.621864 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:43:38.630166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.621771 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:38.638800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.638753 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-86a40d\"/\"test-model-86a40d-kserve-self-signed-certs\"" Oct 31 03:43:38.665613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.665527 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-86a40d\"/\"openshift-service-ca.crt\"" Oct 31 03:43:38.667461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.667437 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-86a40d\"/\"kube-root-ca.crt\"" Oct 31 03:43:38.681236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.679161 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-86a40d\"/\"default-dockercfg-8ck4c\"" Oct 31 03:43:38.835227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.830334 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.863403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.849340 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:43:38.919649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.916340 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.942099 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.948803 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tls-certs\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.949027 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh2gb\" (UniqueName: \"kubernetes.io/projected/276cbeb2-96f4-43d7-ad78-e77a0323f96a-kube-api-access-gh2gb\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.949127 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-model-cache\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.949217 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-home\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.949356 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-dshm\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.949386 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tmp-dir\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.890249 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:38.969913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.962857 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:43:39.000241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:38.986328 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:39.110260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.106799 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": context deadline exceeded" Oct 31 03:43:39.156557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.156517 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tls-certs\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.160044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.158077 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh2gb\" (UniqueName: \"kubernetes.io/projected/276cbeb2-96f4-43d7-ad78-e77a0323f96a-kube-api-access-gh2gb\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.160044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.158190 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-model-cache\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.160044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.158272 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-home\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.160044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.158436 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-dshm\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.160044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.158479 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tmp-dir\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.214269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.179315 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:39.250320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.240547 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tls-certs\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.250566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.250338 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:39.252065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.252039 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:43:39.272574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.272514 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-model-cache\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.277234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.272953 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-home\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.277428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.273608 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tmp-dir\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.323391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.323354 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh2gb\" (UniqueName: \"kubernetes.io/projected/276cbeb2-96f4-43d7-ad78-e77a0323f96a-kube-api-access-gh2gb\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.324709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.324685 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:43:39.339715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.339690 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:43:39.356240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.341223 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:43:39.436228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.435060 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-dshm\") pod \"test-model-86a40d-kserve-6dbbdb988f-kflsp\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.518276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.514807 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:39.556612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.556560 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:39.563885 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:43:39.561452 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:43:39.566422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.566398 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:43:39.953669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.952173 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:43:39.959161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:39.957862 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:43:40.464989 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:43:40.464948 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod276cbeb2_96f4_43d7_ad78_e77a0323f96a.slice/crio-f9973ab2904af555b48e082a03dedf81d82bd82eba648f495f75b65430f1e144 WatchSource:0}: Error finding container f9973ab2904af555b48e082a03dedf81d82bd82eba648f495f75b65430f1e144: Status 404 returned error can't find the container with id f9973ab2904af555b48e082a03dedf81d82bd82eba648f495f75b65430f1e144 Oct 31 03:43:40.465475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:40.465170 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp"] Oct 31 03:43:40.603229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:40.599566 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" event={"ID":"276cbeb2-96f4-43d7-ad78-e77a0323f96a","Type":"ContainerStarted","Data":"f9973ab2904af555b48e082a03dedf81d82bd82eba648f495f75b65430f1e144"} Oct 31 03:43:41.283771 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:43:41.283729 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ec6ff4_0c20_48bd_bce8_4e7377bc0073.slice/crio-b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:43:41.528962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.528882 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:43:41.554886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.554847 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-kube-api-access-b2jl2\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-kube-api-access-b2jl2\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.554913 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-token\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-token\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.554984 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-socket\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555145 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-podinfo\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555234 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-envoy\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555314 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-data\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555353 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-socket" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:41.555507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555413 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istiod-ca-cert\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555464 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-certs\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.555616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555526 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-credential-socket\") pod \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\" (UID: \"09ec6ff4-0c20-48bd-bce8-4e7377bc0073\") " Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.555946 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-certs" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556023 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-credential-socket" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556044 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istiod-ca-cert" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556154 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-data" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556671 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556695 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556709 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556722 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.557219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.556735 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.560303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.559440 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-envoy" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:41.565223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.563709 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-kube-api-access-b2jl2" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "kube-api-access-b2jl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:41.569243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.568600 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-token" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:41.569243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.568695 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-podinfo" pod "09ec6ff4-0c20-48bd-bce8-4e7377bc0073" (UID: "09ec6ff4-0c20-48bd-bce8-4e7377bc0073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:43:41.598148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.598108 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb"] Oct 31 03:43:41.599595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.599569 5520 cpu_manager.go:414] "containerMap: removing container" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" Oct 31 03:43:41.599595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.599595 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" Oct 31 03:43:41.599827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.599813 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerName="istio-proxy" Oct 31 03:43:41.627146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.627111 5520 generic.go:356] "Generic (PLEG): container finished" podID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" containerID="b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24" exitCode=0 Oct 31 03:43:41.627358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.627339 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" Oct 31 03:43:41.636920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.635364 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.636920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.636353 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" event={"ID":"276cbeb2-96f4-43d7-ad78-e77a0323f96a","Type":"ContainerStarted","Data":"05d68fad773c334bdd1f0e9b0ef116c675b1ce9fc7daeb18bb142f6126bc644b"} Oct 31 03:43:41.636920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.636410 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb"] Oct 31 03:43:41.636920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.636488 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" event={"ID":"09ec6ff4-0c20-48bd-bce8-4e7377bc0073","Type":"ContainerDied","Data":"b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24"} Oct 31 03:43:41.636920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.636517 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7" event={"ID":"09ec6ff4-0c20-48bd-bce8-4e7377bc0073","Type":"ContainerDied","Data":"32f4ff9c8bc84bfd9ad2f646da31cc3a169b8304e091b912b004f0eae1b87820"} Oct 31 03:43:41.636920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.636579 5520 scope.go:122] "RemoveContainer" containerID="b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24" Oct 31 03:43:41.652373 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.651027 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-034fe8-openshift-default-dockercfg-vbg8p\"" Oct 31 03:43:41.663370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.661243 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podStartSLOduration=3.661224495 podStartE2EDuration="3.661224495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:41.654529126 +0000 UTC m=+1645.849052270" watchObservedRunningTime="2026-10-31 03:43:41.661224495 +0000 UTC m=+1645.855747612" Oct 31 03:43:41.668454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668419 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d9602-8bbe-49f3-a808-97c335b0b1d8-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668509 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668536 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668678 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668703 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvlbz\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-kube-api-access-nvlbz\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668737 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668768 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668838 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.668956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.668935 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.669069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.669054 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2jl2\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-kube-api-access-b2jl2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.669133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.669077 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.669133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.669091 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.669133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.669108 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ec6ff4-0c20-48bd-bce8-4e7377bc0073-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:41.675712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.675686 5520 scope.go:122] "RemoveContainer" containerID="b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24" Oct 31 03:43:41.676083 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:43:41.676057 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24\": container with ID starting with b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24 not found: ID does not exist" containerID="b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24" Oct 31 03:43:41.676186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.676096 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24"} err="failed to get container status \"b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24\": rpc error: code = NotFound desc = could not find container \"b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24\": container with ID starting with b87fcff8deba7436a00633d1b5f9d00f457ea2fc6af51f43d94e6e7cafad4d24 not found: ID does not exist" Oct 31 03:43:41.716514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.716473 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7"] Oct 31 03:43:41.721493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.721463 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-h25b7"] Oct 31 03:43:41.786248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.786118 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d9602-8bbe-49f3-a808-97c335b0b1d8-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.786438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.786338 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d9602-8bbe-49f3-a808-97c335b0b1d8-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.786521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.786487 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.786898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.786832 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.787482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.787435 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.787638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.786519 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.787961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.787940 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.788459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.788430 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvlbz\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-kube-api-access-nvlbz\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.788981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.788364 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.789472 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.788919 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.789635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.789617 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.790101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.790082 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.790822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.790803 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.791543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.791427 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.799340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.798419 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.800033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.799979 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.881395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.880853 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvlbz\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-kube-api-access-nvlbz\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.900265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.900164 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:41.921546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.921513 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ec6ff4-0c20-48bd-bce8-4e7377bc0073" path="/var/lib/kubelet/pods/09ec6ff4-0c20-48bd-bce8-4e7377bc0073/volumes" Oct 31 03:43:41.975268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:41.975224 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:42.299837 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:43:42.299722 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004d9602_8bbe_49f3_a808_97c335b0b1d8.slice/crio-cff2349dd93a24aa19881d5c509e7ccaddc20e592adfaa36600be0b90371c2ef WatchSource:0}: Error finding container cff2349dd93a24aa19881d5c509e7ccaddc20e592adfaa36600be0b90371c2ef: Status 404 returned error can't find the container with id cff2349dd93a24aa19881d5c509e7ccaddc20e592adfaa36600be0b90371c2ef Oct 31 03:43:42.299837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.299787 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb"] Oct 31 03:43:42.304618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.304580 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:42.305161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.304669 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:42.305161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.304709 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:42.537226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.537003 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:42.644482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.644388 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" event={"ID":"004d9602-8bbe-49f3-a808-97c335b0b1d8","Type":"ContainerStarted","Data":"8781a06db00ee77c66d14a701d775f0d0991383bb2a8bff3d560e6c670206406"} Oct 31 03:43:42.644482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.644452 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" event={"ID":"004d9602-8bbe-49f3-a808-97c335b0b1d8","Type":"ContainerStarted","Data":"cff2349dd93a24aa19881d5c509e7ccaddc20e592adfaa36600be0b90371c2ef"} Oct 31 03:43:42.739260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.726380 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podStartSLOduration=36.726356884 podStartE2EDuration="36.726356884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:42.700687567 +0000 UTC m=+1646.895210724" watchObservedRunningTime="2026-10-31 03:43:42.726356884 +0000 UTC m=+1646.920880007" Oct 31 03:43:42.975770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:42.975685 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:44.113227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:44.111364 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 03:43:44.113227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:44.111438 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 03:43:44.979046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:44.978602 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 03:43:44.979046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:44.978664 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 03:43:45.976837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:45.976791 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 03:43:45.977516 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:45.977464 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 03:43:46.977329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:46.976525 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 03:43:46.977329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:46.976585 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 03:43:47.979135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:47.978656 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 03:43:47.979135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:47.978723 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 03:43:48.876308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.874874 5520 generic.go:356] "Generic (PLEG): container finished" podID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerID="30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c" exitCode=0 Oct 31 03:43:48.876308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.875138 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" event={"ID":"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf","Type":"ContainerDied","Data":"30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c"} Oct 31 03:43:48.876308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.875777 5520 scope.go:122] "RemoveContainer" containerID="30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c" Oct 31 03:43:48.880571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.879698 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:48.880571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.879788 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:48.880571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.879838 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:48.978229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.971077 5520 generic.go:356] "Generic (PLEG): container finished" podID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerID="7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a" exitCode=0 Oct 31 03:43:48.978229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.971217 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" event={"ID":"7e0d8f04-2f35-4581-a012-9e205b8ced0e","Type":"ContainerDied","Data":"7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a"} Oct 31 03:43:48.978229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.971858 5520 scope.go:122] "RemoveContainer" containerID="7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a" Oct 31 03:43:48.984903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.983483 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 31 03:43:48.984903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:48.983532 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 31 03:43:49.056137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:49.052586 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:49.056137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:49.052716 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:49.056137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:49.052765 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:43:49.523812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:49.523752 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:49.539517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:49.539350 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:50.034844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.034809 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" event={"ID":"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf","Type":"ContainerStarted","Data":"4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73"} Oct 31 03:43:50.039273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.039237 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" event={"ID":"7e0d8f04-2f35-4581-a012-9e205b8ced0e","Type":"ContainerStarted","Data":"15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181"} Oct 31 03:43:50.066049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.066009 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:50.067642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.067616 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:50.087264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.076535 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:43:50.087264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.076850 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:43:50.292288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.284600 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:43:50.338226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.306532 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:43:50.338523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.338489 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:43:50.707241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.698513 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:43:50.707241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.698651 5520 patch_prober.go:29] interesting pod/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:43:50.707241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:50.698693 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:43:51.297663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:51.272470 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:43:51.297663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:51.272531 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:43:51.386088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:51.385998 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:43:51.694619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:51.694579 5520 patch_prober.go:29] interesting pod/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:43:51.694898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:51.694868 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:43:52.274001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:52.273545 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:43:52.274001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:52.273617 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:43:52.704226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:52.694950 5520 patch_prober.go:29] interesting pod/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:43:52.704226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:52.695012 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:43:53.293230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:53.288530 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:43:53.293230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:53.288590 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:43:53.695641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:53.695592 5520 patch_prober.go:29] interesting pod/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:43:53.695941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:53.695911 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:43:54.290230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:54.283607 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:43:54.290230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:54.283670 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:43:54.692718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:54.692684 5520 patch_prober.go:29] interesting pod/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:43:54.693007 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:54.692980 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:43:54.728384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:54.728350 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:43:55.285642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:55.285608 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:43:55.285962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:55.285936 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:43:55.694649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:55.694608 5520 patch_prober.go:29] interesting pod/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 31 03:43:55.695333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:55.695301 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 31 03:43:56.274490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:56.274062 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 31 03:43:56.274490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:56.274120 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 31 03:43:56.699083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:56.699047 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:43:56.700506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:56.700479 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:43:57.278369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:57.278339 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:43:57.280429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:43:57.280403 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:44:02.728613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.728571 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs"] Oct 31 03:44:02.749749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.749714 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.794550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.767715 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs"] Oct 31 03:44:02.802404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.780128 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-034fe8\"/\"kube-root-ca.crt\"" Oct 31 03:44:02.802630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.780136 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-034fe8\"/\"default-dockercfg-gv8ls\"" Oct 31 03:44:02.802785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.780236 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-034fe8\"/\"openshift-service-ca.crt\"" Oct 31 03:44:02.802889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.780263 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-034fe8\"/\"test-model-034fe8-kserve-self-signed-certs\"" Oct 31 03:44:02.870560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.870522 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl59d\" (UniqueName: \"kubernetes.io/projected/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-kube-api-access-vl59d\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.870763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.870566 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-model-cache\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.870817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.870777 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tmp-dir\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.870940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.870915 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-dshm\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.871009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.870994 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-home\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.871166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.871146 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tls-certs\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.993287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.989395 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tls-certs\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.993287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.989739 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl59d\" (UniqueName: \"kubernetes.io/projected/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-kube-api-access-vl59d\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.993287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.989769 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-model-cache\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.993287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.989899 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tmp-dir\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.993287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.989994 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-dshm\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.993287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.990038 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-home\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:02.993287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.990560 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-home\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:03.006240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:02.999257 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tmp-dir\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:03.006240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:03.000050 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-model-cache\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:03.021545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:03.021510 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tls-certs\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:03.046060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:03.046028 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl59d\" (UniqueName: \"kubernetes.io/projected/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-kube-api-access-vl59d\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:03.055249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:03.048533 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-dshm\") pod \"test-model-034fe8-kserve-67cfc86bf9-rrsrs\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:03.093228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:03.092286 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:03.769152 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:44:03.769120 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e5ffadc_df64_44f8_a33c_bb9c2960ca0c.slice/crio-00d135756c1a0b2ba53bf8bb99398a56413051979bd9ada5dbbed6cb5fc84eaa WatchSource:0}: Error finding container 00d135756c1a0b2ba53bf8bb99398a56413051979bd9ada5dbbed6cb5fc84eaa: Status 404 returned error can't find the container with id 00d135756c1a0b2ba53bf8bb99398a56413051979bd9ada5dbbed6cb5fc84eaa Oct 31 03:44:03.770153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:03.770129 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs"] Oct 31 03:44:04.443292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:04.443248 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" event={"ID":"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c","Type":"ContainerStarted","Data":"a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f"} Oct 31 03:44:04.443484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:04.443303 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" event={"ID":"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c","Type":"ContainerStarted","Data":"00d135756c1a0b2ba53bf8bb99398a56413051979bd9ada5dbbed6cb5fc84eaa"} Oct 31 03:44:04.465798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:04.465747 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podStartSLOduration=2.46573294 podStartE2EDuration="2.46573294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:04.463327825 +0000 UTC m=+1668.657850967" watchObservedRunningTime="2026-10-31 03:44:04.46573294 +0000 UTC m=+1668.660256061" Oct 31 03:44:05.110429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:05.110385 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:06.125105 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:06.125066 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7"] Oct 31 03:44:06.125637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:06.125319 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" containerID="cri-o://517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115" gracePeriod=30 Oct 31 03:44:06.462496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:06.462460 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d"] Oct 31 03:44:06.462710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:06.462690 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" containerID="cri-o://13c9dbb17fbd0fcd409b96501b0e5240552dcf791420161c40914cfe807013f2" gracePeriod=30 Oct 31 03:44:08.876269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.876236 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452"] Oct 31 03:44:08.881399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.881378 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:08.904027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.903977 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452"] Oct 31 03:44:08.904395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.904373 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46ebfe\"/\"openshift-service-ca.crt\"" Oct 31 03:44:08.904563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.904476 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46ebfe\"/\"kube-root-ca.crt\"" Oct 31 03:44:08.904617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.904574 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46ebfe\"/\"default-dockercfg-j8xb2\"" Oct 31 03:44:08.905440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.905416 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46ebfe\"/\"test-model-46ebfe-kserve-self-signed-certs\"" Oct 31 03:44:08.936502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.936431 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-tmp-dir\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:08.936665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.936529 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-model-cache\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:08.936665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.936560 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47568df-5e8e-477e-a372-dd8710ab78d9-tls-certs\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:08.936665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.936631 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-dshm\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:08.936989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.936971 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-home\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:08.937070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:08.937052 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmn2\" (UniqueName: \"kubernetes.io/projected/d47568df-5e8e-477e-a372-dd8710ab78d9-kube-api-access-6cmn2\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.038975 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.038941 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-home\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039014 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmn2\" (UniqueName: \"kubernetes.io/projected/d47568df-5e8e-477e-a372-dd8710ab78d9-kube-api-access-6cmn2\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039141 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-tmp-dir\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039165 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-model-cache\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039191 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47568df-5e8e-477e-a372-dd8710ab78d9-tls-certs\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039309 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-dshm\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039441 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-home\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039558 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-model-cache\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.039982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.039952 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-tmp-dir\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.042873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.042843 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47568df-5e8e-477e-a372-dd8710ab78d9-tls-certs\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.043118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.043080 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-dshm\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.048172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.048147 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmn2\" (UniqueName: \"kubernetes.io/projected/d47568df-5e8e-477e-a372-dd8710ab78d9-kube-api-access-6cmn2\") pod \"test-model-46ebfe-kserve-68466fbfbc-7d452\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.254961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.253719 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:09.528541 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:09.528495 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:44:09.836416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:09.836365 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452"] Oct 31 03:44:09.853435 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:44:09.849230 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd47568df_5e8e_477e_a372_dd8710ab78d9.slice/crio-5b0d20af8e99c7d8e59f230f29458c66c61501598552cc50a14ba35ac1af3d1f WatchSource:0}: Error finding container 5b0d20af8e99c7d8e59f230f29458c66c61501598552cc50a14ba35ac1af3d1f: Status 404 returned error can't find the container with id 5b0d20af8e99c7d8e59f230f29458c66c61501598552cc50a14ba35ac1af3d1f Oct 31 03:44:10.592229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:10.591756 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" event={"ID":"d47568df-5e8e-477e-a372-dd8710ab78d9","Type":"ContainerStarted","Data":"9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9"} Oct 31 03:44:10.592229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:10.591811 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" event={"ID":"d47568df-5e8e-477e-a372-dd8710ab78d9","Type":"ContainerStarted","Data":"5b0d20af8e99c7d8e59f230f29458c66c61501598552cc50a14ba35ac1af3d1f"} Oct 31 03:44:10.637810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:10.637757 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podStartSLOduration=2.637740679 podStartE2EDuration="2.637740679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:10.617647827 +0000 UTC m=+1674.812170989" watchObservedRunningTime="2026-10-31 03:44:10.637740679 +0000 UTC m=+1674.832263800" Oct 31 03:44:11.258386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.258357 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:11.283704 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:11.283667 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372fbb32_2eb1_4f98_b790_2a93d25d08e9.slice/crio-conmon-517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:44:11.525524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.525483 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:44:11.618736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.618427 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c"] Oct 31 03:44:11.620145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.620115 5520 cpu_manager.go:414] "containerMap: removing container" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" Oct 31 03:44:11.620145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.620146 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" Oct 31 03:44:11.620458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.620441 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerName="istio-proxy" Oct 31 03:44:11.649935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.649903 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c"] Oct 31 03:44:11.650098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.650040 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.651630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.651599 5520 generic.go:356] "Generic (PLEG): container finished" podID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" containerID="517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115" exitCode=0 Oct 31 03:44:11.651760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.651726 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" event={"ID":"372fbb32-2eb1-4f98-b790-2a93d25d08e9","Type":"ContainerDied","Data":"517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115"} Oct 31 03:44:11.651822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.651783 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" event={"ID":"372fbb32-2eb1-4f98-b790-2a93d25d08e9","Type":"ContainerDied","Data":"b9476bcec9d85be6240deade3c1fbbe96c5ce191a50389a6810ca4f137a0ef1a"} Oct 31 03:44:11.651873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.651822 5520 scope.go:122] "RemoveContainer" containerID="517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115" Oct 31 03:44:11.652047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.652028 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7" Oct 31 03:44:11.658752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.658626 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b68780-openshift-default-dockercfg-7824b\"" Oct 31 03:44:11.659542 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.659490 5520 generic.go:356] "Generic (PLEG): container finished" podID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerID="13c9dbb17fbd0fcd409b96501b0e5240552dcf791420161c40914cfe807013f2" exitCode=0 Oct 31 03:44:11.660260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.660191 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" event={"ID":"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd","Type":"ContainerDied","Data":"13c9dbb17fbd0fcd409b96501b0e5240552dcf791420161c40914cfe807013f2"} Oct 31 03:44:11.713944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.713920 5520 scope.go:122] "RemoveContainer" containerID="517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115" Oct 31 03:44:11.714418 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:11.714388 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115\": container with ID starting with 517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115 not found: ID does not exist" containerID="517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115" Oct 31 03:44:11.714535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.714425 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115"} err="failed to get container status \"517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115\": rpc error: code = NotFound desc = could not find container \"517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115\": container with ID starting with 517e911b9fda9e3c83264af5e94f7bf50638ca0fb6e30697ccb0fcc3240c1115 not found: ID does not exist" Oct 31 03:44:11.725294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.725194 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-credential-socket\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.725498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.725457 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-credential-socket" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.725739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.725708 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istiod-ca-cert\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.725957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.725937 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-kube-api-access-rpvtz\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-kube-api-access-rpvtz\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.726104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.726083 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istiod-ca-cert" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:11.726328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.726313 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-token\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-token\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.726434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.726420 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-socket\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.726571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.726556 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-data\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.726835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.726817 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-envoy\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.727055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.726818 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-socket" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.727055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.726861 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-data" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.727235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.727222 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-podinfo\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.727366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.727357 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-certs\") pod \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\" (UID: \"372fbb32-2eb1-4f98-b790-2a93d25d08e9\") " Oct 31 03:44:11.727671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.727636 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-certs" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.727781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.727688 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.727843 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58l7\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-kube-api-access-m58l7\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.727945 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728216 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728311 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728375 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728404 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728621 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728722 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728854 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728872 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728887 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728904 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.729616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.728917 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.733787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.733764 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-kube-api-access-rpvtz" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "kube-api-access-rpvtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.733899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.733780 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-podinfo" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:11.733957 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.733833 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-envoy" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.736246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.735705 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-token" pod "372fbb32-2eb1-4f98-b790-2a93d25d08e9" (UID: "372fbb32-2eb1-4f98-b790-2a93d25d08e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.830432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.830394 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.830625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.830493 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.830856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.830832 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.830931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.830905 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.830993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.830935 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.831152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.831131 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.831313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.831291 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.831395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.831364 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.831579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.831532 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58l7\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-kube-api-access-m58l7\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.831648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.831608 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.831708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.831650 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.832063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.832041 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.832402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.832381 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.833046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.833026 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpvtz\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-kube-api-access-rpvtz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.834814 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.834787 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.835745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.835706 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.835867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.835754 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.835867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.835771 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372fbb32-2eb1-4f98-b790-2a93d25d08e9-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.835996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.835870 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.835996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.835967 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.840016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.839990 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.842173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.842127 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58l7\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-kube-api-access-m58l7\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:11.865303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.865281 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:44:11.931473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.931446 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995"] Oct 31 03:44:11.932579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.932555 5520 cpu_manager.go:414] "containerMap: removing container" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" Oct 31 03:44:11.932579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.932579 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" Oct 31 03:44:11.932843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.932827 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" containerName="istio-proxy" Oct 31 03:44:11.938173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938068 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-data\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.938363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938171 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-kube-api-access-n66k5\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-kube-api-access-n66k5\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.938363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938252 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-envoy\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.938363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938332 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-certs\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.938546 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938448 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-token\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-token\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.938606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938547 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istiod-ca-cert\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.938665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938626 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-podinfo\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.938718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.938659 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-socket\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.944665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.939020 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-credential-socket\") pod \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\" (UID: \"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd\") " Oct 31 03:44:11.944665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.939061 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:11.944665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.940749 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istiod-ca-cert" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:11.948423 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.941351 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-credential-socket" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.948423 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.943396 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-data" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.948601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.948578 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-socket" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.948722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.948673 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-envoy" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.949763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.949736 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-certs" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.953574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.953520 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-kube-api-access-n66k5" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "kube-api-access-n66k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.955441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.955360 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-token" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.955663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.955632 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-podinfo" pod "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" (UID: "88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:11.967490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:11.966389 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995"] Oct 31 03:44:12.023885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.023848 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:12.028164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.028138 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7"] Oct 31 03:44:12.033676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.033652 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-l9lf7"] Oct 31 03:44:12.050903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.050872 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69df727-93b9-498f-8ff0-07845d18ceea-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.051146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.051124 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.052041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.052016 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69df727-93b9-498f-8ff0-07845d18ceea-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.052284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.052266 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.053770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.053738 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.053894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.053874 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.054060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054035 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.054127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054094 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.054164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054123 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2gls\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-kube-api-access-z2gls\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.054416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054400 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054421 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n66k5\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-kube-api-access-n66k5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054438 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054452 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054468 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054481 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054496 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054511 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.054747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.054528 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.156858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.156822 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.156904 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.156932 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.156951 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2gls\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-kube-api-access-z2gls\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.157061 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69df727-93b9-498f-8ff0-07845d18ceea-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.157092 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.157149 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69df727-93b9-498f-8ff0-07845d18ceea-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.157257 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.157356 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.157599 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.157936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.157636 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.158175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.158150 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.158308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.158295 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69df727-93b9-498f-8ff0-07845d18ceea-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.158547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.158511 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.160848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.160497 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.160848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.160815 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69df727-93b9-498f-8ff0-07845d18ceea-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.171610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.171542 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2gls\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-kube-api-access-z2gls\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.180684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.180507 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.277695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.277658 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:12.374885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.374841 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:12.375023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.374942 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:12.375023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.374986 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:12.394156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.394039 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c"] Oct 31 03:44:12.623659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.623629 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995"] Oct 31 03:44:12.627374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.626781 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:12.627374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.626842 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:12.627374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.626880 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:12.681246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.678294 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" event={"ID":"88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd","Type":"ContainerDied","Data":"59d6f3456c50d5be6c01ff9a3bdcceb9b45c896d96c4ff20ee80fb1c608c3448"} Oct 31 03:44:12.681246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.678367 5520 scope.go:122] "RemoveContainer" containerID="13c9dbb17fbd0fcd409b96501b0e5240552dcf791420161c40914cfe807013f2" Oct 31 03:44:12.681246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.678500 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d" Oct 31 03:44:12.701682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.700552 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" event={"ID":"c69df727-93b9-498f-8ff0-07845d18ceea","Type":"ContainerStarted","Data":"4f783d7da9969353cb2be5b562f2d9b664ec24a44507f4116c04d701cecad24a"} Oct 31 03:44:12.704715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.704692 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" event={"ID":"33e15ae6-cdb8-46aa-b7b8-983b7c19920c","Type":"ContainerStarted","Data":"03e588934e172c2c07286643fbcf4ea37bc5984af5978292ad545b1397442455"} Oct 31 03:44:12.704918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.704899 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" event={"ID":"33e15ae6-cdb8-46aa-b7b8-983b7c19920c","Type":"ContainerStarted","Data":"1ece5f157c891ddfbf116fdc42ffed34799cdb3d48931cc2c0312882f34a1656"} Oct 31 03:44:12.750928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.750867 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podStartSLOduration=29.750846987 podStartE2EDuration="29.750846987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:12.743768378 +0000 UTC m=+1676.938291517" watchObservedRunningTime="2026-10-31 03:44:12.750846987 +0000 UTC m=+1676.945370110" Oct 31 03:44:12.766799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.766768 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d"] Oct 31 03:44:12.774141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:12.774112 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-dm95d"] Oct 31 03:44:13.026969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.026940 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:13.027326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.027303 5520 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:44:13.027445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.027350 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:44:13.095412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.095379 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:13.114077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.114047 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:13.719526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.719488 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" event={"ID":"c69df727-93b9-498f-8ff0-07845d18ceea","Type":"ContainerStarted","Data":"b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d"} Oct 31 03:44:13.732707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.732679 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:44:13.747100 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.747040 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podStartSLOduration=25.747026237 podStartE2EDuration="25.747026237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:13.740538335 +0000 UTC m=+1677.935061458" watchObservedRunningTime="2026-10-31 03:44:13.747026237 +0000 UTC m=+1677.941549408" Oct 31 03:44:13.925435 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.925398 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372fbb32-2eb1-4f98-b790-2a93d25d08e9" path="/var/lib/kubelet/pods/372fbb32-2eb1-4f98-b790-2a93d25d08e9/volumes" Oct 31 03:44:13.926106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:13.926084 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd" path="/var/lib/kubelet/pods/88fa6e14-b6a5-4c6a-85c3-0006c0e67cbd/volumes" Oct 31 03:44:14.030109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:14.030027 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:14.281600 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:14.281508 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:14.303431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:14.303371 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 03:44:14.303758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:14.303710 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 03:44:14.793229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:14.789245 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:14.793229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:14.792922 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:44:15.294834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:15.294795 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 03:44:15.295154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:15.295123 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 03:44:16.288282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.288248 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 03:44:16.288926 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.288892 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 03:44:16.657353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.657256 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr"] Oct 31 03:44:16.680250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.671453 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:16.680250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.673987 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr"] Oct 31 03:44:16.723174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.716074 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b68780-dockercfg-4cn5v\"" Oct 31 03:44:16.745552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.730886 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6ab6d-8790-4555-961c-9a00d02bc19a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:16.745552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.731309 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgbw\" (UniqueName: \"kubernetes.io/projected/7ed6ab6d-8790-4555-961c-9a00d02bc19a-kube-api-access-rxgbw\") pod \"payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:16.762226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.760471 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b68780\"" Oct 31 03:44:16.843229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.841818 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6ab6d-8790-4555-961c-9a00d02bc19a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:16.843229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:16.842130 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgbw\" (UniqueName: \"kubernetes.io/projected/7ed6ab6d-8790-4555-961c-9a00d02bc19a-kube-api-access-rxgbw\") pod \"payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:17.233240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:17.230655 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6ab6d-8790-4555-961c-9a00d02bc19a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:17.352260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:17.303368 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgbw\" (UniqueName: \"kubernetes.io/projected/7ed6ab6d-8790-4555-961c-9a00d02bc19a-kube-api-access-rxgbw\") pod \"payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:17.352260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:17.315398 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 03:44:17.352260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:17.315471 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 03:44:17.352260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:17.325482 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:17.754961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:17.754923 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr"] Oct 31 03:44:17.766427 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:44:17.766333 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed6ab6d_8790_4555_961c_9a00d02bc19a.slice/crio-ba95003c8571c7e8abbb7b487305b9806cec792569e4bcd5a4137a1acfc720f7 WatchSource:0}: Error finding container ba95003c8571c7e8abbb7b487305b9806cec792569e4bcd5a4137a1acfc720f7: Status 404 returned error can't find the container with id ba95003c8571c7e8abbb7b487305b9806cec792569e4bcd5a4137a1acfc720f7 Oct 31 03:44:18.285229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:18.278251 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 03:44:18.285229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:18.278310 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 03:44:18.403230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:18.392775 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" event={"ID":"7ed6ab6d-8790-4555-961c-9a00d02bc19a","Type":"ContainerStarted","Data":"6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37"} Oct 31 03:44:18.403230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:18.392867 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" event={"ID":"7ed6ab6d-8790-4555-961c-9a00d02bc19a","Type":"ContainerStarted","Data":"ba95003c8571c7e8abbb7b487305b9806cec792569e4bcd5a4137a1acfc720f7"} Oct 31 03:44:18.403230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:18.394442 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:18.439225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:18.438636 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" podStartSLOduration=2.438614623 podStartE2EDuration="2.438614623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:18.42059853 +0000 UTC m=+1682.615121676" watchObservedRunningTime="2026-10-31 03:44:18.438614623 +0000 UTC m=+1682.633137747" Oct 31 03:44:19.267243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:19.266596 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:19.289224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:19.283722 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 03:44:19.289224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:19.283779 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 03:44:19.650060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:19.649980 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:19.708225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:19.705839 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:44:20.291311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:20.288884 5520 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 03:44:20.291311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:20.288948 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 03:44:21.298295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:21.298257 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:21.302221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:21.299357 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:21.302221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:21.300905 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:44:21.996826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:21.996794 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv"] Oct 31 03:44:21.997044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:21.997020 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" containerID="cri-o://69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d" gracePeriod=30 Oct 31 03:44:27.401042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.401011 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:44:27.433253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.433222 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq"] Oct 31 03:44:27.434341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.434319 5520 cpu_manager.go:414] "containerMap: removing container" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" Oct 31 03:44:27.434341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.434339 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" Oct 31 03:44:27.434633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.434617 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d903c2-8f71-46a3-8523-930b333f870c" containerName="istio-proxy" Oct 31 03:44:27.438743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.438721 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.454624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.454599 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-86a40d-openshift-default-dockercfg-49mvt\"" Oct 31 03:44:27.471637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.463116 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq"] Oct 31 03:44:27.551474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.551441 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19d903c2-8f71-46a3-8523-930b333f870c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d903c2-8f71-46a3-8523-930b333f870c-istio-podinfo\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.551678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.551500 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-istio-token\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-istio-token\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.551678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.551562 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-envoy\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.551818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.551757 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-certs\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.551818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.551796 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d903c2-8f71-46a3-8523-930b333f870c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d903c2-8f71-46a3-8523-930b333f870c-istiod-ca-cert\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.551937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.551833 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-kube-api-access-zj8n2\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-kube-api-access-zj8n2\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.551995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.551937 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-data\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.552184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552162 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-credential-socket\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.552297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552194 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d903c2-8f71-46a3-8523-930b333f870c-istiod-ca-cert" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:27.552297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552261 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-socket\") pod \"19d903c2-8f71-46a3-8523-930b333f870c\" (UID: \"19d903c2-8f71-46a3-8523-930b333f870c\") " Oct 31 03:44:27.552688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552412 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-data" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.552688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552634 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-credential-socket" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.552869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552731 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-socket" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.552930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552860 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.552930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552878 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-certs" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.552930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.552909 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chph7\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-kube-api-access-chph7\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.553156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.553124 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.553395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.553370 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.553593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.553558 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ab80f9-6359-4b2e-bea4-2b047417170b-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.553681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.553636 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.553743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.553689 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.553887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.553818 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.554047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.554023 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.554515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.554492 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.554601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.554521 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d903c2-8f71-46a3-8523-930b333f870c-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.554601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.554537 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.554601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.554553 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.554601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.554570 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.554955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.554926 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-envoy" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.555679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.555656 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-kube-api-access-zj8n2" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "kube-api-access-zj8n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:27.556183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.556159 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19d903c2-8f71-46a3-8523-930b333f870c-istio-podinfo" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:27.568222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.561361 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-istio-token" pod "19d903c2-8f71-46a3-8523-930b333f870c" (UID: "19d903c2-8f71-46a3-8523-930b333f870c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.660680 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.660736 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chph7\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-kube-api-access-chph7\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.660845 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.660988 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661102 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ab80f9-6359-4b2e-bea4-2b047417170b-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661153 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661188 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661293 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661364 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661673 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d903c2-8f71-46a3-8523-930b333f870c-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661692 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661706 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d903c2-8f71-46a3-8523-930b333f870c-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.662877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.661723 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8n2\" (UniqueName: \"kubernetes.io/projected/19d903c2-8f71-46a3-8523-930b333f870c-kube-api-access-zj8n2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.681502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.666932 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.681502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.670794 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.681502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.674520 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.681502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.679545 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.681502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.679853 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.681502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.680957 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.692225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.681906 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ab80f9-6359-4b2e-bea4-2b047417170b-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.692225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.690149 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chph7\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-kube-api-access-chph7\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.737639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.737609 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.752073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.750877 5520 generic.go:356] "Generic (PLEG): container finished" podID="19d903c2-8f71-46a3-8523-930b333f870c" containerID="69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d" exitCode=0 Oct 31 03:44:27.752073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.751319 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" event={"ID":"19d903c2-8f71-46a3-8523-930b333f870c","Type":"ContainerDied","Data":"69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d"} Oct 31 03:44:27.752073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.751393 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" event={"ID":"19d903c2-8f71-46a3-8523-930b333f870c","Type":"ContainerDied","Data":"f48c5228f913f4346a3e945b4fefce35bdd7a35662a3e2e944aca6d4bf3ab590"} Oct 31 03:44:27.752073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.751599 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv" Oct 31 03:44:27.752073 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.751682 5520 scope.go:122] "RemoveContainer" containerID="69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d" Oct 31 03:44:27.765437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.765409 5520 scope.go:122] "RemoveContainer" containerID="69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d" Oct 31 03:44:27.765795 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:27.765769 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d\": container with ID starting with 69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d not found: ID does not exist" containerID="69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d" Oct 31 03:44:27.765898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.765806 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d"} err="failed to get container status \"69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d\": rpc error: code = NotFound desc = could not find container \"69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d\": container with ID starting with 69e2fdd55fbba4bb3c5462afe8442b9da2c3e96d24038dfc6b08c8a6b576bc5d not found: ID does not exist" Oct 31 03:44:27.777442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.777417 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:27.787477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.787453 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv"] Oct 31 03:44:27.793297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.793274 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-c7vjv"] Oct 31 03:44:27.928928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:27.928902 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d903c2-8f71-46a3-8523-930b333f870c" path="/var/lib/kubelet/pods/19d903c2-8f71-46a3-8523-930b333f870c/volumes" Oct 31 03:44:28.079566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.079536 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq"] Oct 31 03:44:28.081463 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:44:28.081437 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ab80f9_6359_4b2e_bea4_2b047417170b.slice/crio-e6c02603dfee5c60f1a31190720ef3f6738336c2f8382b1dace42b727954bc76 WatchSource:0}: Error finding container e6c02603dfee5c60f1a31190720ef3f6738336c2f8382b1dace42b727954bc76: Status 404 returned error can't find the container with id e6c02603dfee5c60f1a31190720ef3f6738336c2f8382b1dace42b727954bc76 Oct 31 03:44:28.085468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.085426 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:28.085579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.085523 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:28.085579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.085567 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:28.763019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.762979 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" event={"ID":"d1ab80f9-6359-4b2e-bea4-2b047417170b","Type":"ContainerStarted","Data":"2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26"} Oct 31 03:44:28.763498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.763034 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" event={"ID":"d1ab80f9-6359-4b2e-bea4-2b047417170b","Type":"ContainerStarted","Data":"e6c02603dfee5c60f1a31190720ef3f6738336c2f8382b1dace42b727954bc76"} Oct 31 03:44:28.777823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.777776 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:28.834573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:28.834523 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podStartSLOduration=1.834508283 podStartE2EDuration="1.834508283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:28.829031982 +0000 UTC m=+1693.023555121" watchObservedRunningTime="2026-10-31 03:44:28.834508283 +0000 UTC m=+1693.029031404" Oct 31 03:44:29.785433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:29.785401 5520 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 03:44:29.786349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:29.786174 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 03:44:30.526468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:30.526222 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:44:30.783298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:30.782775 5520 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 03:44:30.783298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:30.782836 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 03:44:31.791229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:31.790630 5520 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 03:44:31.791229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:31.790691 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 03:44:32.783354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:32.783316 5520 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 03:44:32.783669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:32.783639 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 03:44:33.784887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:33.784835 5520 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 03:44:33.785699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:33.785658 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 03:44:34.777856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:34.777817 5520 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 03:44:34.778087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:34.777882 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 03:44:35.783159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:35.783122 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:35.783626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:35.783574 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:35.784535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:35.784512 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:44:39.361481 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:39.361442 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:44:44.585333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:44.585287 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2"] Oct 31 03:44:44.585783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:44.585527 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" containerID="cri-o://edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c" gracePeriod=30 Oct 31 03:44:49.991178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:49.991150 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:44:50.034645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.034609 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j"] Oct 31 03:44:50.035985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.035961 5520 cpu_manager.go:414] "containerMap: removing container" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" Oct 31 03:44:50.036088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.035989 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" Oct 31 03:44:50.036301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.036283 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" containerName="istio-proxy" Oct 31 03:44:50.041173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.041153 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.051439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.051405 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j"] Oct 31 03:44:50.165421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165387 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-kube-api-access-9fhf9\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-kube-api-access-9fhf9\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.165638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165435 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-envoy\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.165638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165477 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-socket\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.165638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165596 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-certs\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.165638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165631 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/026a2997-9eea-48fc-87a9-3e255f11746f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a2997-9eea-48fc-87a9-3e255f11746f-istio-podinfo\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.165893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165713 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-credential-socket\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.165893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165796 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026a2997-9eea-48fc-87a9-3e255f11746f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a2997-9eea-48fc-87a9-3e255f11746f-istiod-ca-cert\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.165893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165795 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-socket" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:50.165893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165845 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-certs" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:50.166109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.165940 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-data\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.166109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.166025 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-istio-token\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-istio-token\") pod \"026a2997-9eea-48fc-87a9-3e255f11746f\" (UID: \"026a2997-9eea-48fc-87a9-3e255f11746f\") " Oct 31 03:44:50.166109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.166027 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-credential-socket" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:50.166462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.166354 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026a2997-9eea-48fc-87a9-3e255f11746f-istiod-ca-cert" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:50.166462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.166422 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.166587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.166528 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-data" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:50.166709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.166687 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.167052 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.167032 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.167234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.167195 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbbq\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-kube-api-access-mnbbq\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.167454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.167440 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.167612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.167598 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.168535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168318 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-kube-api-access-9fhf9" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "kube-api-access-9fhf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:50.168535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168423 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.168535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168472 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-envoy" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:50.168535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168528 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168579 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168795 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fhf9\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-kube-api-access-9fhf9\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168821 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168836 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168850 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168865 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168878 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a2997-9eea-48fc-87a9-3e255f11746f-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.169021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.168892 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a2997-9eea-48fc-87a9-3e255f11746f-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.169500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.169484 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-istio-token" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:50.169966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.169942 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/026a2997-9eea-48fc-87a9-3e255f11746f-istio-podinfo" pod "026a2997-9eea-48fc-87a9-3e255f11746f" (UID: "026a2997-9eea-48fc-87a9-3e255f11746f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:50.219429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.219339 5520 generic.go:356] "Generic (PLEG): container finished" podID="026a2997-9eea-48fc-87a9-3e255f11746f" containerID="edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c" exitCode=0 Oct 31 03:44:50.220008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.219431 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" event={"ID":"026a2997-9eea-48fc-87a9-3e255f11746f","Type":"ContainerDied","Data":"edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c"} Oct 31 03:44:50.220254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.220233 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" event={"ID":"026a2997-9eea-48fc-87a9-3e255f11746f","Type":"ContainerDied","Data":"50362b0b3f2a5d017cec5a893808f22c6bf15945e41bdbf82c84a112263346ca"} Oct 31 03:44:50.220469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.220359 5520 scope.go:122] "RemoveContainer" containerID="edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c" Oct 31 03:44:50.220603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.219481 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2" Oct 31 03:44:50.235772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.235753 5520 scope.go:122] "RemoveContainer" containerID="edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c" Oct 31 03:44:50.236096 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:50.236071 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c\": container with ID starting with edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c not found: ID does not exist" containerID="edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c" Oct 31 03:44:50.236192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.236103 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c"} err="failed to get container status \"edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c\": rpc error: code = NotFound desc = could not find container \"edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c\": container with ID starting with edca6783c69b22cefbd938f3df39649e2c6ae0a0f7d64dba50639d7f0d821b8c not found: ID does not exist" Oct 31 03:44:50.249402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.248580 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2"] Oct 31 03:44:50.254428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.254407 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-77vz2"] Oct 31 03:44:50.270152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270125 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.270478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270455 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.270657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270638 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.270705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270684 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbbq\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-kube-api-access-mnbbq\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.270799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270783 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.270856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270813 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.270920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270854 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.270920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270882 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.271033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270942 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.271033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.270992 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.271155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.271137 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.271227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.271153 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a2997-9eea-48fc-87a9-3e255f11746f-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.271227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.271171 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a2997-9eea-48fc-87a9-3e255f11746f-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:50.271335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.271288 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.271418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.271397 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.271801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.271726 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.273369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.273350 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.273452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.273400 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.281355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.281329 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbbq\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-kube-api-access-mnbbq\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.283585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.283558 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.396225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.384063 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:50.768834 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:44:50.768798 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e3fdd4_9152_4bb8_b98d_2445eb671f4b.slice/crio-44e872a11a560110bdc6386cf9ef386e01093faff0614a4031d687f2c0ee3dcf WatchSource:0}: Error finding container 44e872a11a560110bdc6386cf9ef386e01093faff0614a4031d687f2c0ee3dcf: Status 404 returned error can't find the container with id 44e872a11a560110bdc6386cf9ef386e01093faff0614a4031d687f2c0ee3dcf Oct 31 03:44:50.769113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.769094 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j"] Oct 31 03:44:50.773350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.773311 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:50.773448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.773398 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:50.773448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:50.773442 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:51.298033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.297988 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" event={"ID":"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b","Type":"ContainerStarted","Data":"8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c"} Oct 31 03:44:51.300299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.300269 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" event={"ID":"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b","Type":"ContainerStarted","Data":"44e872a11a560110bdc6386cf9ef386e01093faff0614a4031d687f2c0ee3dcf"} Oct 31 03:44:51.353761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.353686 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podStartSLOduration=90.353664313 podStartE2EDuration="1m30.353664313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:51.349999279 +0000 UTC m=+1715.544522401" watchObservedRunningTime="2026-10-31 03:44:51.353664313 +0000 UTC m=+1715.548187436" Oct 31 03:44:51.386178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.386092 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:51.386382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.386228 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 03:44:51.386382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.386270 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 03:44:51.727628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.727598 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh"] Oct 31 03:44:51.727830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:51.727790 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" containerID="cri-o://ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973" gracePeriod=30 Oct 31 03:44:52.022281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:52.014442 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026a2997-9eea-48fc-87a9-3e255f11746f" path="/var/lib/kubelet/pods/026a2997-9eea-48fc-87a9-3e255f11746f/volumes" Oct 31 03:44:52.022281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:52.014996 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv"] Oct 31 03:44:52.022281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:52.015310 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" containerID="cri-o://bfa17b7e94d07ad543e20d1bbdb5e059da5a46606aff8967c8e383e05b9ba984" gracePeriod=30 Oct 31 03:44:52.400382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:52.388320 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 03:44:52.400382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:52.388384 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 03:44:53.396737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:53.392284 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 03:44:53.396737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:53.392336 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 03:44:54.385239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:54.384730 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 03:44:54.385239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:54.384789 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 03:44:55.388282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:55.385629 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 03:44:55.388282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:55.385702 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 03:44:56.394224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:56.385600 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 31 03:44:56.394224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:56.385679 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 31 03:44:57.241764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.241738 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:44:57.294225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.294165 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn"] Oct 31 03:44:57.298224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.295750 5520 cpu_manager.go:414] "containerMap: removing container" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" Oct 31 03:44:57.298224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.295775 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" Oct 31 03:44:57.298224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.296019 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerName="istio-proxy" Oct 31 03:44:57.312227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.300859 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.312227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.304982 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b68780-serving-cert\"" Oct 31 03:44:57.317226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.315417 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn"] Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365359 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-credential-socket\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365497 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-certs\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365550 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-socket\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365653 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-credential-socket" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365705 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-data\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365787 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-socket" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365788 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-kube-api-access-8pn5v\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-kube-api-access-8pn5v\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365830 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-certs" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365877 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-token\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-token\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.365922 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-podinfo\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.366008 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-envoy\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.366008 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-data" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.366057 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istiod-ca-cert\") pod \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\" (UID: \"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd\") " Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.366379 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-maas-api-tls\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.366462 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.366720 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfkj\" (UniqueName: \"kubernetes.io/projected/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-kube-api-access-msfkj\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.366933 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-metrics-certs\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.367062 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.367081 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.367091 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.367225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.367101 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.368474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.367261 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istiod-ca-cert" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:57.371451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.368886 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-podinfo" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:57.435320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.413396 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-token" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:57.435320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.413547 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-kube-api-access-8pn5v" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "kube-api-access-8pn5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:57.435320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.427423 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-envoy" pod "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" (UID: "48dcd49d-aab0-41df-bcf7-a7e9bbc788fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:57.494506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.485803 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:57.494506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.489266 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:57.496156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.496086 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:44:57.533085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.533047 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfkj\" (UniqueName: \"kubernetes.io/projected/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-kube-api-access-msfkj\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.533379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.533356 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-metrics-certs\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.533492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.533476 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-maas-api-tls\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.533575 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.533555 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.533965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.533946 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pn5v\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-kube-api-access-8pn5v\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.534004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.533972 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.534004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.533987 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.534004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.534001 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.534106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.534014 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:57.535951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.535916 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.540383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.540357 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-metrics-certs\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.542621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.542596 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfkj\" (UniqueName: \"kubernetes.io/projected/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-kube-api-access-msfkj\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.542743 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.542721 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-maas-api-tls\") pod \"maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.544342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.544323 5520 generic.go:356] "Generic (PLEG): container finished" podID="794074d9-ca09-408d-be79-9ebd51d93c81" containerID="bfa17b7e94d07ad543e20d1bbdb5e059da5a46606aff8967c8e383e05b9ba984" exitCode=0 Oct 31 03:44:57.544511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.544391 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" event={"ID":"794074d9-ca09-408d-be79-9ebd51d93c81","Type":"ContainerDied","Data":"bfa17b7e94d07ad543e20d1bbdb5e059da5a46606aff8967c8e383e05b9ba984"} Oct 31 03:44:57.546419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.546402 5520 generic.go:356] "Generic (PLEG): container finished" podID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" containerID="ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973" exitCode=0 Oct 31 03:44:57.546503 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.546492 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" Oct 31 03:44:57.548285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.546437 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" event={"ID":"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd","Type":"ContainerDied","Data":"ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973"} Oct 31 03:44:57.548375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.548309 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh" event={"ID":"48dcd49d-aab0-41df-bcf7-a7e9bbc788fd","Type":"ContainerDied","Data":"30e7876e3f1f9d533214c4a495ff2e8b332a9df1c4cadeb9bbbcd148b12f19ba"} Oct 31 03:44:57.548375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.548334 5520 scope.go:122] "RemoveContainer" containerID="ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973" Oct 31 03:44:57.566711 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.566682 5520 scope.go:122] "RemoveContainer" containerID="ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973" Oct 31 03:44:57.579260 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:57.567366 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973\": container with ID starting with ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973 not found: ID does not exist" containerID="ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973" Oct 31 03:44:57.579260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.567409 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973"} err="failed to get container status \"ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973\": rpc error: code = NotFound desc = could not find container \"ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973\": container with ID starting with ce12dea208758529ccd2f15f60c5241579ddde3d32c225f0f872f3e68ee60973 not found: ID does not exist" Oct 31 03:44:57.586250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.584920 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh"] Oct 31 03:44:57.607517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.605991 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-hbvkh"] Oct 31 03:44:57.635654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.635616 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:57.923627 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.923597 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48dcd49d-aab0-41df-bcf7-a7e9bbc788fd" path="/var/lib/kubelet/pods/48dcd49d-aab0-41df-bcf7-a7e9bbc788fd/volumes" Oct 31 03:44:57.950405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.950368 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:44:57.955467 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:44:57.955440 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea1f583_9a3f_47ae_a2bf_23847dfa9a54.slice/crio-958b3bc3cf9156f1c8be92297dd1aa02b050210e37485e3964a8b13b34130b14 WatchSource:0}: Error finding container 958b3bc3cf9156f1c8be92297dd1aa02b050210e37485e3964a8b13b34130b14: Status 404 returned error can't find the container with id 958b3bc3cf9156f1c8be92297dd1aa02b050210e37485e3964a8b13b34130b14 Oct 31 03:44:57.957484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.957456 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn"] Oct 31 03:44:57.995144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.995054 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq"] Oct 31 03:44:57.996902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.996723 5520 cpu_manager.go:414] "containerMap: removing container" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" Oct 31 03:44:57.996902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.996749 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" Oct 31 03:44:57.997148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:57.996967 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" containerName="istio-proxy" Oct 31 03:44:58.004434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.002730 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.010769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.009105 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq"] Oct 31 03:44:58.042817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.042789 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/794074d9-ca09-408d-be79-9ebd51d93c81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794074d9-ca09-408d-be79-9ebd51d93c81-istio-podinfo\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.042842 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-istio-token\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-istio-token\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.042884 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-envoy\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.042986 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-certs\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.043049 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-data\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.043123 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794074d9-ca09-408d-be79-9ebd51d93c81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794074d9-ca09-408d-be79-9ebd51d93c81-istiod-ca-cert\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.043160 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-credential-socket\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.043356 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-socket\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.043401 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-kube-api-access-g2798\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-kube-api-access-g2798\") pod \"794074d9-ca09-408d-be79-9ebd51d93c81\" (UID: \"794074d9-ca09-408d-be79-9ebd51d93c81\") " Oct 31 03:44:58.043846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.043814 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794074d9-ca09-408d-be79-9ebd51d93c81-istiod-ca-cert" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:58.044432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.044400 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-data" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:58.045146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.045116 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-credential-socket" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:58.045450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.045428 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.045871 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.045841 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794074d9-ca09-408d-be79-9ebd51d93c81-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.046076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.046056 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-certs" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:58.046137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.046093 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-socket" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:58.047714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.047685 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-kube-api-access-g2798" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "kube-api-access-g2798". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:58.047808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.047727 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-istio-token" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:58.053306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.053277 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/794074d9-ca09-408d-be79-9ebd51d93c81-istio-podinfo" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:58.053306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.053297 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-envoy" pod "794074d9-ca09-408d-be79-9ebd51d93c81" (UID: "794074d9-ca09-408d-be79-9ebd51d93c81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.195901 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.195970 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.196123 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.196151 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.196291 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5z7z\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-kube-api-access-w5z7z\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.196340 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de000444-58e9-4de9-b340-4b8ec8b9d583-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.196373 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.196454 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.197320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.196541 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.201227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.200786 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.201227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.200817 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.201227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.200832 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2798\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-kube-api-access-g2798\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.201227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.200847 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794074d9-ca09-408d-be79-9ebd51d93c81-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.201227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.200861 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794074d9-ca09-408d-be79-9ebd51d93c81-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.201227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.200874 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.201227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.200891 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794074d9-ca09-408d-be79-9ebd51d93c81-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:58.302589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302503 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.302589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302553 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.302825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302668 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.302825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302696 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.302825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302811 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5z7z\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-kube-api-access-w5z7z\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.302998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302862 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de000444-58e9-4de9-b340-4b8ec8b9d583-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.302998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302897 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.302998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.302984 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.303170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.303007 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.303170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.303052 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.303170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.303087 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.303415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.303393 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.303563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.303536 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.303868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.303848 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de000444-58e9-4de9-b340-4b8ec8b9d583-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.305420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.305395 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.305945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.305929 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.311187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.311162 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5z7z\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-kube-api-access-w5z7z\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.311534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.311518 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.337841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.337817 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:44:58.557648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.557547 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" event={"ID":"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54","Type":"ContainerStarted","Data":"24238d1c7c42861303fb9f18163fc82713be300bf934e11d082921522b0a79fe"} Oct 31 03:44:58.557648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.557619 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:44:58.557648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.557646 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" event={"ID":"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54","Type":"ContainerStarted","Data":"958b3bc3cf9156f1c8be92297dd1aa02b050210e37485e3964a8b13b34130b14"} Oct 31 03:44:58.563032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.563005 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" Oct 31 03:44:58.573157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.572820 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv" event={"ID":"794074d9-ca09-408d-be79-9ebd51d93c81","Type":"ContainerDied","Data":"409f3a61811f13594243a9f3835cdea17ffca87278a0d0f4bf2b2b81c52b2607"} Oct 31 03:44:58.573157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.572891 5520 scope.go:122] "RemoveContainer" containerID="bfa17b7e94d07ad543e20d1bbdb5e059da5a46606aff8967c8e383e05b9ba984" Oct 31 03:44:58.580467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.580360 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" podStartSLOduration=20.144381063 podStartE2EDuration="20.580333477s" totalImagesPullingTime="435.952414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:58.577574695 +0000 UTC m=+1722.772097833" watchObservedRunningTime="2026-10-31 03:44:58.580333477 +0000 UTC m=+1722.774856600" Oct 31 03:44:58.610269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.600821 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv"] Oct 31 03:44:58.610269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.609120 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-vllzv"] Oct 31 03:44:58.632535 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:58.632491 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794074d9_ca09_408d_be79_9ebd51d93c81.slice/crio-409f3a61811f13594243a9f3835cdea17ffca87278a0d0f4bf2b2b81c52b2607\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794074d9_ca09_408d_be79_9ebd51d93c81.slice\": RecentStats: unable to find data in memory cache]" Oct 31 03:44:58.642870 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:44:58.642833 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794074d9_ca09_408d_be79_9ebd51d93c81.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794074d9_ca09_408d_be79_9ebd51d93c81.slice/crio-409f3a61811f13594243a9f3835cdea17ffca87278a0d0f4bf2b2b81c52b2607\": RecentStats: unable to find data in memory cache]" Oct 31 03:44:58.661458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.661434 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq"] Oct 31 03:44:58.662311 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:44:58.662288 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde000444_58e9_4de9_b340_4b8ec8b9d583.slice/crio-0918313846dad315a8e9cb54545b264bacab1664b968c3130a7108e88e3f48a6 WatchSource:0}: Error finding container 0918313846dad315a8e9cb54545b264bacab1664b968c3130a7108e88e3f48a6: Status 404 returned error can't find the container with id 0918313846dad315a8e9cb54545b264bacab1664b968c3130a7108e88e3f48a6 Oct 31 03:44:58.665550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.665519 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:58.665635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.665585 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:58.665635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:58.665616 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:44:59.660360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:59.660323 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" event={"ID":"de000444-58e9-4de9-b340-4b8ec8b9d583","Type":"ContainerStarted","Data":"f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9"} Oct 31 03:44:59.660904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:59.660874 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" event={"ID":"de000444-58e9-4de9-b340-4b8ec8b9d583","Type":"ContainerStarted","Data":"0918313846dad315a8e9cb54545b264bacab1664b968c3130a7108e88e3f48a6"} Oct 31 03:44:59.692366 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:59.692309 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podStartSLOduration=98.692290545 podStartE2EDuration="1m38.692290545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:59.682849382 +0000 UTC m=+1723.877372514" watchObservedRunningTime="2026-10-31 03:44:59.692290545 +0000 UTC m=+1723.886813668" Oct 31 03:44:59.923062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:44:59.922985 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794074d9-ca09-408d-be79-9ebd51d93c81" path="/var/lib/kubelet/pods/794074d9-ca09-408d-be79-9ebd51d93c81/volumes" Oct 31 03:45:00.153730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.153691 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj"] Oct 31 03:45:00.160018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.159989 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9"] Oct 31 03:45:00.160187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.160102 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.165407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.165379 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph"] Oct 31 03:45:00.165849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.165828 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.184183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.184147 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:45:00.184183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.184151 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.223061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.223032 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9"] Oct 31 03:45:00.233428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.233401 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph"] Oct 31 03:45:00.242519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.241743 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj"] Oct 31 03:45:00.308706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.308662 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e702e1d3-994f-4787-84ab-e4c991329ef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.310229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.310182 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefaabdf-e4ac-405b-a776-5b5137710477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.310596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.310568 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4bd\" (UniqueName: \"kubernetes.io/projected/cefaabdf-e4ac-405b-a776-5b5137710477-kube-api-access-fz4bd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.310804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.310783 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa0ef49-ec77-405b-907f-f30d5878209e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.312451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.310896 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgmx\" (UniqueName: \"kubernetes.io/projected/e702e1d3-994f-4787-84ab-e4c991329ef2-kube-api-access-qqgmx\") pod \"maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.312451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.312430 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfq6j\" (UniqueName: \"kubernetes.io/projected/afa0ef49-ec77-405b-907f-f30d5878209e-kube-api-access-zfq6j\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.338288 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.338251 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:45:00.339774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.339748 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 03:45:00.339904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.339869 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.485644 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefaabdf-e4ac-405b-a776-5b5137710477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.485766 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4bd\" (UniqueName: \"kubernetes.io/projected/cefaabdf-e4ac-405b-a776-5b5137710477-kube-api-access-fz4bd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.486002 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa0ef49-ec77-405b-907f-f30d5878209e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.486097 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgmx\" (UniqueName: \"kubernetes.io/projected/e702e1d3-994f-4787-84ab-e4c991329ef2-kube-api-access-qqgmx\") pod \"maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.486136 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfq6j\" (UniqueName: \"kubernetes.io/projected/afa0ef49-ec77-405b-907f-f30d5878209e-kube-api-access-zfq6j\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.486295 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e702e1d3-994f-4787-84ab-e4c991329ef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.487302 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e702e1d3-994f-4787-84ab-e4c991329ef2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.500287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.488041 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefaabdf-e4ac-405b-a776-5b5137710477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.525599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.525408 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa0ef49-ec77-405b-907f-f30d5878209e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.531953 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.531893 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgmx\" (UniqueName: \"kubernetes.io/projected/e702e1d3-994f-4787-84ab-e4c991329ef2-kube-api-access-qqgmx\") pod \"maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.532884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.532849 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4bd\" (UniqueName: \"kubernetes.io/projected/cefaabdf-e4ac-405b-a776-5b5137710477-kube-api-access-fz4bd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.533140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.533109 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfq6j\" (UniqueName: \"kubernetes.io/projected/afa0ef49-ec77-405b-907f-f30d5878209e-kube-api-access-zfq6j\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.554320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.554283 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:00.561156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.561128 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:00.615791 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.615763 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:00.956641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:00.956605 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9"] Oct 31 03:45:00.959032 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:45:00.959003 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode702e1d3_994f_4787_84ab_e4c991329ef2.slice/crio-991301f992bdb03f2810b409bdababdcc32df8936354d0192c80dc275dfbb430 WatchSource:0}: Error finding container 991301f992bdb03f2810b409bdababdcc32df8936354d0192c80dc275dfbb430: Status 404 returned error can't find the container with id 991301f992bdb03f2810b409bdababdcc32df8936354d0192c80dc275dfbb430 Oct 31 03:45:01.025884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.020843 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph"] Oct 31 03:45:01.216667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.205476 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj"] Oct 31 03:45:01.339946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.339917 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 03:45:01.340087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.339966 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 03:45:01.753225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.753160 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" event={"ID":"cefaabdf-e4ac-405b-a776-5b5137710477","Type":"ContainerStarted","Data":"23919bf46a807feef8b96a2d3cc7720162621ebd5b6d24ffdb3081c0b65a138f"} Oct 31 03:45:01.754949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.754911 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerStarted","Data":"3932e11e568bf2c26273a6dca1161c4101aec0f671f70b120481afcf2316c7e5"} Oct 31 03:45:01.755105 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.754965 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerStarted","Data":"991301f992bdb03f2810b409bdababdcc32df8936354d0192c80dc275dfbb430"} Oct 31 03:45:01.756748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.756719 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerStarted","Data":"7836bb2c6e881b5654ea75689029f9744bc165c300aee33ac92da5fa6a7955a2"} Oct 31 03:45:01.756856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.756770 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerStarted","Data":"005458150fd8cedf0896f2ba6dd4718af11010ac813e978c599668b73c51ee2d"} Oct 31 03:45:01.828072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:01.828010 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" podStartSLOduration=1.827995891 podStartE2EDuration="1.827995891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.806495736 +0000 UTC m=+1726.001018857" watchObservedRunningTime="2026-10-31 03:45:01.827995891 +0000 UTC m=+1726.022519031" Oct 31 03:45:02.338639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.338297 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 03:45:02.338639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.338355 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 03:45:02.774230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.771436 5520 generic.go:356] "Generic (PLEG): container finished" podID="cefaabdf-e4ac-405b-a776-5b5137710477" containerID="e42f387d7da873f0ccd5d24b8e4908c5996145e0c7c4cab5017846f2cb6691bf" exitCode=60 Oct 31 03:45:02.774230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.771632 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" event={"ID":"cefaabdf-e4ac-405b-a776-5b5137710477","Type":"ContainerDied","Data":"e42f387d7da873f0ccd5d24b8e4908c5996145e0c7c4cab5017846f2cb6691bf"} Oct 31 03:45:02.788225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.783081 5520 generic.go:356] "Generic (PLEG): container finished" podID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerID="3932e11e568bf2c26273a6dca1161c4101aec0f671f70b120481afcf2316c7e5" exitCode=7 Oct 31 03:45:02.788225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.787373 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerDied","Data":"3932e11e568bf2c26273a6dca1161c4101aec0f671f70b120481afcf2316c7e5"} Oct 31 03:45:02.792260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.790317 5520 scope.go:122] "RemoveContainer" containerID="e42f387d7da873f0ccd5d24b8e4908c5996145e0c7c4cab5017846f2cb6691bf" Oct 31 03:45:02.792260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.790949 5520 scope.go:122] "RemoveContainer" containerID="3932e11e568bf2c26273a6dca1161c4101aec0f671f70b120481afcf2316c7e5" Oct 31 03:45:02.809410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.809348 5520 generic.go:356] "Generic (PLEG): container finished" podID="afa0ef49-ec77-405b-907f-f30d5878209e" containerID="7836bb2c6e881b5654ea75689029f9744bc165c300aee33ac92da5fa6a7955a2" exitCode=60 Oct 31 03:45:02.809410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.809409 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerDied","Data":"7836bb2c6e881b5654ea75689029f9744bc165c300aee33ac92da5fa6a7955a2"} Oct 31 03:45:02.810570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.810543 5520 scope.go:122] "RemoveContainer" containerID="7836bb2c6e881b5654ea75689029f9744bc165c300aee33ac92da5fa6a7955a2" Oct 31 03:45:02.811727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:02.811683 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" podStartSLOduration=2.811669684 podStartE2EDuration="2.811669684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.824809504 +0000 UTC m=+1726.019332624" watchObservedRunningTime="2026-10-31 03:45:02.811669684 +0000 UTC m=+1727.006192805" Oct 31 03:45:03.338466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:03.338434 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 03:45:03.338660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:03.338484 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 03:45:03.821981 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:03.821894 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerStarted","Data":"1fa8828e201675fa5d133637eb91eb4d4f8175e4a1ea779042fe7d69181889fa"} Oct 31 03:45:03.825596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:03.825562 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerStarted","Data":"8961b02b2b6096cb79703c7a5b03ff4e6bab3d7d9b0398c3143e90c96ab3d3f6"} Oct 31 03:45:03.828247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:03.828219 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" event={"ID":"cefaabdf-e4ac-405b-a776-5b5137710477","Type":"ContainerStarted","Data":"32d082e4ee359fd6052b7ff71d0c41ccacfa23fadfe974365591984ad031a737"} Oct 31 03:45:03.856447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:03.856387 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" podStartSLOduration=3.856372097 podStartE2EDuration="3.856372097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:03.851824181 +0000 UTC m=+1728.046347302" watchObservedRunningTime="2026-10-31 03:45:03.856372097 +0000 UTC m=+1728.050895217" Oct 31 03:45:04.351275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.346844 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 03:45:04.351275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.346903 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 03:45:04.695287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.675049 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:45:04.864376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.864339 5520 generic.go:356] "Generic (PLEG): container finished" podID="afa0ef49-ec77-405b-907f-f30d5878209e" containerID="8961b02b2b6096cb79703c7a5b03ff4e6bab3d7d9b0398c3143e90c96ab3d3f6" exitCode=60 Oct 31 03:45:04.864680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.864409 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerDied","Data":"8961b02b2b6096cb79703c7a5b03ff4e6bab3d7d9b0398c3143e90c96ab3d3f6"} Oct 31 03:45:04.864680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.864565 5520 scope.go:122] "RemoveContainer" containerID="7836bb2c6e881b5654ea75689029f9744bc165c300aee33ac92da5fa6a7955a2" Oct 31 03:45:04.864848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.864832 5520 scope.go:122] "RemoveContainer" containerID="8961b02b2b6096cb79703c7a5b03ff4e6bab3d7d9b0398c3143e90c96ab3d3f6" Oct 31 03:45:04.865270 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:04.865248 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph_odh-ai-gateway-infra(afa0ef49-ec77-405b-907f-f30d5878209e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" Oct 31 03:45:04.908918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.908876 5520 generic.go:356] "Generic (PLEG): container finished" podID="cefaabdf-e4ac-405b-a776-5b5137710477" containerID="32d082e4ee359fd6052b7ff71d0c41ccacfa23fadfe974365591984ad031a737" exitCode=60 Oct 31 03:45:04.909135 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.909094 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" event={"ID":"cefaabdf-e4ac-405b-a776-5b5137710477","Type":"ContainerDied","Data":"32d082e4ee359fd6052b7ff71d0c41ccacfa23fadfe974365591984ad031a737"} Oct 31 03:45:04.909697 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.909674 5520 scope.go:122] "RemoveContainer" containerID="32d082e4ee359fd6052b7ff71d0c41ccacfa23fadfe974365591984ad031a737" Oct 31 03:45:04.911048 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:04.911016 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj_odh-ai-gateway-infra(cefaabdf-e4ac-405b-a776-5b5137710477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" Oct 31 03:45:04.922842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.922815 5520 generic.go:356] "Generic (PLEG): container finished" podID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerID="1fa8828e201675fa5d133637eb91eb4d4f8175e4a1ea779042fe7d69181889fa" exitCode=7 Oct 31 03:45:04.923132 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.923112 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerDied","Data":"1fa8828e201675fa5d133637eb91eb4d4f8175e4a1ea779042fe7d69181889fa"} Oct 31 03:45:04.923718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.923692 5520 scope.go:122] "RemoveContainer" containerID="1fa8828e201675fa5d133637eb91eb4d4f8175e4a1ea779042fe7d69181889fa" Oct 31 03:45:04.924193 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:04.924167 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9_odh-ai-gateway-infra(e702e1d3-994f-4787-84ab-e4c991329ef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" Oct 31 03:45:04.926857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.926832 5520 scope.go:122] "RemoveContainer" containerID="e42f387d7da873f0ccd5d24b8e4908c5996145e0c7c4cab5017846f2cb6691bf" Oct 31 03:45:04.943305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:04.943273 5520 scope.go:122] "RemoveContainer" containerID="3932e11e568bf2c26273a6dca1161c4101aec0f671f70b120481afcf2316c7e5" Oct 31 03:45:05.338560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:05.338470 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 03:45:05.338560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:05.338528 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 03:45:05.933066 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:05.933036 5520 scope.go:122] "RemoveContainer" containerID="8961b02b2b6096cb79703c7a5b03ff4e6bab3d7d9b0398c3143e90c96ab3d3f6" Oct 31 03:45:05.933579 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:05.933530 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph_odh-ai-gateway-infra(afa0ef49-ec77-405b-907f-f30d5878209e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" Oct 31 03:45:05.934874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:05.934847 5520 scope.go:122] "RemoveContainer" containerID="32d082e4ee359fd6052b7ff71d0c41ccacfa23fadfe974365591984ad031a737" Oct 31 03:45:05.935365 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:05.935337 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj_odh-ai-gateway-infra(cefaabdf-e4ac-405b-a776-5b5137710477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" Oct 31 03:45:05.937389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:05.937359 5520 scope.go:122] "RemoveContainer" containerID="1fa8828e201675fa5d133637eb91eb4d4f8175e4a1ea779042fe7d69181889fa" Oct 31 03:45:05.937672 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:05.937654 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9_odh-ai-gateway-infra(e702e1d3-994f-4787-84ab-e4c991329ef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" Oct 31 03:45:06.390069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:06.389518 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:45:06.392013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:06.391991 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:45:06.392992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:06.392975 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:45:09.362734 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:09.362694 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:45:14.909973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:14.909937 5520 scope.go:122] "RemoveContainer" containerID="32d082e4ee359fd6052b7ff71d0c41ccacfa23fadfe974365591984ad031a737" Oct 31 03:45:15.947255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:15.946894 5520 scope.go:122] "RemoveContainer" containerID="1fa8828e201675fa5d133637eb91eb4d4f8175e4a1ea779042fe7d69181889fa" Oct 31 03:45:15.952228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:15.948061 5520 scope.go:122] "RemoveContainer" containerID="8961b02b2b6096cb79703c7a5b03ff4e6bab3d7d9b0398c3143e90c96ab3d3f6" Oct 31 03:45:16.074270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:16.074195 5520 generic.go:356] "Generic (PLEG): container finished" podID="cefaabdf-e4ac-405b-a776-5b5137710477" containerID="872e514409086d712a4177a8457715f51d1204b10fffc24307ec22b43cab90f1" exitCode=60 Oct 31 03:45:16.074270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:16.074247 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" event={"ID":"cefaabdf-e4ac-405b-a776-5b5137710477","Type":"ContainerDied","Data":"872e514409086d712a4177a8457715f51d1204b10fffc24307ec22b43cab90f1"} Oct 31 03:45:16.074507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:16.074345 5520 scope.go:122] "RemoveContainer" containerID="32d082e4ee359fd6052b7ff71d0c41ccacfa23fadfe974365591984ad031a737" Oct 31 03:45:16.074905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:16.074881 5520 scope.go:122] "RemoveContainer" containerID="872e514409086d712a4177a8457715f51d1204b10fffc24307ec22b43cab90f1" Oct 31 03:45:16.075453 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:16.075428 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj_odh-ai-gateway-infra(cefaabdf-e4ac-405b-a776-5b5137710477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" Oct 31 03:45:17.088886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.088842 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerStarted","Data":"4ea11de809c8bf3e01b006f43fec50800e213afd4df41ddb48d537116b873403"} Oct 31 03:45:17.091126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.091103 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerStarted","Data":"1abdc0c91dc2f692335e9a1b7110eee4504239ae0a9a2a6cbf406728e04f4b80"} Oct 31 03:45:17.108964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.108929 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj"] Oct 31 03:45:17.541860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.541832 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:17.586577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.586544 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x"] Oct 31 03:45:17.587590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.587567 5520 cpu_manager.go:414] "containerMap: removing container" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:17.587590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.587590 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:17.587796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.587655 5520 cpu_manager.go:414] "containerMap: removing container" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:17.587796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.587663 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:17.587917 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.587900 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:17.587982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.587926 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:17.600508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.600467 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x"] Oct 31 03:45:17.600904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.600830 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:17.751272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.745765 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefaabdf-e4ac-405b-a776-5b5137710477-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefaabdf-e4ac-405b-a776-5b5137710477-maas-api-service-ca\") pod \"cefaabdf-e4ac-405b-a776-5b5137710477\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " Oct 31 03:45:17.751272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.746105 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefaabdf-e4ac-405b-a776-5b5137710477-kube-api-access-fz4bd\" (UniqueName: \"kubernetes.io/projected/cefaabdf-e4ac-405b-a776-5b5137710477-kube-api-access-fz4bd\") pod \"cefaabdf-e4ac-405b-a776-5b5137710477\" (UID: \"cefaabdf-e4ac-405b-a776-5b5137710477\") " Oct 31 03:45:17.751272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.746909 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45b2a27-9d07-4cde-ba44-8409b01e8057-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:17.751272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.747116 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpc2\" (UniqueName: \"kubernetes.io/projected/e45b2a27-9d07-4cde-ba44-8409b01e8057-kube-api-access-qmpc2\") pod \"maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:17.774585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.761458 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefaabdf-e4ac-405b-a776-5b5137710477-maas-api-service-ca" pod "cefaabdf-e4ac-405b-a776-5b5137710477" (UID: "cefaabdf-e4ac-405b-a776-5b5137710477"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:17.774585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.768822 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefaabdf-e4ac-405b-a776-5b5137710477-kube-api-access-fz4bd" pod "cefaabdf-e4ac-405b-a776-5b5137710477" (UID: "cefaabdf-e4ac-405b-a776-5b5137710477"). InnerVolumeSpecName "kube-api-access-fz4bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:17.925225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.924111 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45b2a27-9d07-4cde-ba44-8409b01e8057-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:17.925225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.924317 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpc2\" (UniqueName: \"kubernetes.io/projected/e45b2a27-9d07-4cde-ba44-8409b01e8057-kube-api-access-qmpc2\") pod \"maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:17.925225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.924535 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefaabdf-e4ac-405b-a776-5b5137710477-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:17.925225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.924555 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4bd\" (UniqueName: \"kubernetes.io/projected/cefaabdf-e4ac-405b-a776-5b5137710477-kube-api-access-fz4bd\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:17.925657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:17.925472 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45b2a27-9d07-4cde-ba44-8409b01e8057-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:18.034305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.033439 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpc2\" (UniqueName: \"kubernetes.io/projected/e45b2a27-9d07-4cde-ba44-8409b01e8057-kube-api-access-qmpc2\") pod \"maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:18.049148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.049118 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:18.104881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.104856 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" Oct 31 03:45:18.105861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.104898 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj" event={"ID":"cefaabdf-e4ac-405b-a776-5b5137710477","Type":"ContainerDied","Data":"23919bf46a807feef8b96a2d3cc7720162621ebd5b6d24ffdb3081c0b65a138f"} Oct 31 03:45:18.105971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.105894 5520 scope.go:122] "RemoveContainer" containerID="872e514409086d712a4177a8457715f51d1204b10fffc24307ec22b43cab90f1" Oct 31 03:45:18.107649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.107622 5520 generic.go:356] "Generic (PLEG): container finished" podID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerID="4ea11de809c8bf3e01b006f43fec50800e213afd4df41ddb48d537116b873403" exitCode=60 Oct 31 03:45:18.107832 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.107812 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerDied","Data":"4ea11de809c8bf3e01b006f43fec50800e213afd4df41ddb48d537116b873403"} Oct 31 03:45:18.109332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.108373 5520 scope.go:122] "RemoveContainer" containerID="4ea11de809c8bf3e01b006f43fec50800e213afd4df41ddb48d537116b873403" Oct 31 03:45:18.109332 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:18.108786 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9_odh-ai-gateway-infra(e702e1d3-994f-4787-84ab-e4c991329ef2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" Oct 31 03:45:18.132473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.131823 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9"] Oct 31 03:45:18.132473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.131855 5520 scope.go:122] "RemoveContainer" containerID="1fa8828e201675fa5d133637eb91eb4d4f8175e4a1ea779042fe7d69181889fa" Oct 31 03:45:18.173813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.173774 5520 generic.go:356] "Generic (PLEG): container finished" podID="afa0ef49-ec77-405b-907f-f30d5878209e" containerID="1abdc0c91dc2f692335e9a1b7110eee4504239ae0a9a2a6cbf406728e04f4b80" exitCode=60 Oct 31 03:45:18.173971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.173885 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerDied","Data":"1abdc0c91dc2f692335e9a1b7110eee4504239ae0a9a2a6cbf406728e04f4b80"} Oct 31 03:45:18.174611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.174582 5520 scope.go:122] "RemoveContainer" containerID="1abdc0c91dc2f692335e9a1b7110eee4504239ae0a9a2a6cbf406728e04f4b80" Oct 31 03:45:18.175404 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:18.175375 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph_odh-ai-gateway-infra(afa0ef49-ec77-405b-907f-f30d5878209e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" Oct 31 03:45:18.194345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.194305 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph"] Oct 31 03:45:18.213452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.212048 5520 scope.go:122] "RemoveContainer" containerID="8961b02b2b6096cb79703c7a5b03ff4e6bab3d7d9b0398c3143e90c96ab3d3f6" Oct 31 03:45:18.429278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.428862 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj"] Oct 31 03:45:18.531229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.530305 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fc5ece-29890305-nvltj"] Oct 31 03:45:18.756138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:18.755648 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x"] Oct 31 03:45:19.192356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.192326 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" event={"ID":"e45b2a27-9d07-4cde-ba44-8409b01e8057","Type":"ContainerStarted","Data":"a55ff9072365ff89d02667e18ac211d0fdc33920dd2c5793be004af0b9b02519"} Oct 31 03:45:19.457184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.452415 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:19.618295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.613800 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa0ef49-ec77-405b-907f-f30d5878209e-kube-api-access-zfq6j\" (UniqueName: \"kubernetes.io/projected/afa0ef49-ec77-405b-907f-f30d5878209e-kube-api-access-zfq6j\") pod \"afa0ef49-ec77-405b-907f-f30d5878209e\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " Oct 31 03:45:19.618295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.614006 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa0ef49-ec77-405b-907f-f30d5878209e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa0ef49-ec77-405b-907f-f30d5878209e-maas-api-service-ca\") pod \"afa0ef49-ec77-405b-907f-f30d5878209e\" (UID: \"afa0ef49-ec77-405b-907f-f30d5878209e\") " Oct 31 03:45:19.618295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.615290 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa0ef49-ec77-405b-907f-f30d5878209e-maas-api-service-ca" pod "afa0ef49-ec77-405b-907f-f30d5878209e" (UID: "afa0ef49-ec77-405b-907f-f30d5878209e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:19.674186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.667848 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa0ef49-ec77-405b-907f-f30d5878209e-kube-api-access-zfq6j" pod "afa0ef49-ec77-405b-907f-f30d5878209e" (UID: "afa0ef49-ec77-405b-907f-f30d5878209e"). InnerVolumeSpecName "kube-api-access-zfq6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.690414 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn"] Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692019 5520 cpu_manager.go:414] "containerMap: removing container" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692043 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692088 5520 cpu_manager.go:414] "containerMap: removing container" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692095 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692115 5520 cpu_manager.go:414] "containerMap: removing container" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692122 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692359 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692376 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692396 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:19.694229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.692408 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:19.743227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.739947 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa0ef49-ec77-405b-907f-f30d5878209e-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:19.743227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.739985 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfq6j\" (UniqueName: \"kubernetes.io/projected/afa0ef49-ec77-405b-907f-f30d5878209e-kube-api-access-zfq6j\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:19.782229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.757417 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:19.997276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.841758 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:19.997276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.843255 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e702e1d3-994f-4787-84ab-e4c991329ef2-kube-api-access-qqgmx\" (UniqueName: \"kubernetes.io/projected/e702e1d3-994f-4787-84ab-e4c991329ef2-kube-api-access-qqgmx\") pod \"e702e1d3-994f-4787-84ab-e4c991329ef2\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " Oct 31 03:45:19.997276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.844063 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e702e1d3-994f-4787-84ab-e4c991329ef2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e702e1d3-994f-4787-84ab-e4c991329ef2-maas-api-service-ca\") pod \"e702e1d3-994f-4787-84ab-e4c991329ef2\" (UID: \"e702e1d3-994f-4787-84ab-e4c991329ef2\") " Oct 31 03:45:19.997276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.844689 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e702e1d3-994f-4787-84ab-e4c991329ef2-maas-api-service-ca" pod "e702e1d3-994f-4787-84ab-e4c991329ef2" (UID: "e702e1d3-994f-4787-84ab-e4c991329ef2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:19.997276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.847657 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e702e1d3-994f-4787-84ab-e4c991329ef2-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:19.997276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.902410 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e702e1d3-994f-4787-84ab-e4c991329ef2-kube-api-access-qqgmx" pod "e702e1d3-994f-4787-84ab-e4c991329ef2" (UID: "e702e1d3-994f-4787-84ab-e4c991329ef2"). InnerVolumeSpecName "kube-api-access-qqgmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:19.997276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:19.955517 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqgmx\" (UniqueName: \"kubernetes.io/projected/e702e1d3-994f-4787-84ab-e4c991329ef2-kube-api-access-qqgmx\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:20.110229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.058612 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec03780-b3bf-4d80-a07e-cce1cc231978-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:20.110229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.058914 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn464\" (UniqueName: \"kubernetes.io/projected/9ec03780-b3bf-4d80-a07e-cce1cc231978-kube-api-access-sn464\") pod \"maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:20.415571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.415526 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn464\" (UniqueName: \"kubernetes.io/projected/9ec03780-b3bf-4d80-a07e-cce1cc231978-kube-api-access-sn464\") pod \"maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:20.416535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.416504 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec03780-b3bf-4d80-a07e-cce1cc231978-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:20.419379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.419354 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec03780-b3bf-4d80-a07e-cce1cc231978-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.495415 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefaabdf-e4ac-405b-a776-5b5137710477" path="/var/lib/kubelet/pods/cefaabdf-e4ac-405b-a776-5b5137710477/volumes" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.496606 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn"] Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.496705 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm"] Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498510 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498530 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498555 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498563 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498637 5520 cpu_manager.go:414] "containerMap: removing container" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498644 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" containerName="cleanup" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498867 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:20.501702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.498887 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:20.504255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.504231 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm"] Oct 31 03:45:20.504384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.504285 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:20.538256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.530544 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463623f-4b59-41e0-a7cf-c5914f027845-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:20.538256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.530813 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lbg\" (UniqueName: \"kubernetes.io/projected/c463623f-4b59-41e0-a7cf-c5914f027845-kube-api-access-42lbg\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:20.538256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.531784 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn464\" (UniqueName: \"kubernetes.io/projected/9ec03780-b3bf-4d80-a07e-cce1cc231978-kube-api-access-sn464\") pod \"maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:20.588108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.588063 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" event={"ID":"e702e1d3-994f-4787-84ab-e4c991329ef2","Type":"ContainerDied","Data":"991301f992bdb03f2810b409bdababdcc32df8936354d0192c80dc275dfbb430"} Oct 31 03:45:20.588954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.588838 5520 scope.go:122] "RemoveContainer" containerID="4ea11de809c8bf3e01b006f43fec50800e213afd4df41ddb48d537116b873403" Oct 31 03:45:20.596300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.590351 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9" Oct 31 03:45:20.608920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.595394 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" Oct 31 03:45:20.608920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.595581 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph" event={"ID":"afa0ef49-ec77-405b-907f-f30d5878209e","Type":"ContainerDied","Data":"005458150fd8cedf0896f2ba6dd4718af11010ac813e978c599668b73c51ee2d"} Oct 31 03:45:20.675768 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.634353 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lbg\" (UniqueName: \"kubernetes.io/projected/c463623f-4b59-41e0-a7cf-c5914f027845-kube-api-access-42lbg\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:20.684224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.676695 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463623f-4b59-41e0-a7cf-c5914f027845-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:20.733236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.730148 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463623f-4b59-41e0-a7cf-c5914f027845-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:20.787371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.778569 5520 generic.go:356] "Generic (PLEG): container finished" podID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerID="a78cd431f20b59751173fd913026aa59c49bd4a6db9c1753a7e3370b18633c31" exitCode=60 Oct 31 03:45:20.787371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.778697 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" event={"ID":"e45b2a27-9d07-4cde-ba44-8409b01e8057","Type":"ContainerDied","Data":"a78cd431f20b59751173fd913026aa59c49bd4a6db9c1753a7e3370b18633c31"} Oct 31 03:45:20.787371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.779238 5520 scope.go:122] "RemoveContainer" containerID="a78cd431f20b59751173fd913026aa59c49bd4a6db9c1753a7e3370b18633c31" Oct 31 03:45:20.807994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.807777 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:20.922661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:20.922630 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lbg\" (UniqueName: \"kubernetes.io/projected/c463623f-4b59-41e0-a7cf-c5914f027845-kube-api-access-42lbg\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:21.030373 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.030330 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph"] Oct 31 03:45:21.050238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.050192 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:21.074281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.074179 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-ab21e4-29890305-lbfph"] Oct 31 03:45:21.164691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.160785 5520 scope.go:122] "RemoveContainer" containerID="1abdc0c91dc2f692335e9a1b7110eee4504239ae0a9a2a6cbf406728e04f4b80" Oct 31 03:45:21.239302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.239261 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9"] Oct 31 03:45:21.471170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.471135 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b68780-29890305-xchz9"] Oct 31 03:45:21.827410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.805380 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": context deadline exceeded" Oct 31 03:45:21.885422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.885366 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:21.912230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.889388 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/health\": context deadline exceeded" Oct 31 03:45:21.912230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.889512 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:21.941877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.941827 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.080064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:21.992850 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.080064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.049366 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": context deadline exceeded" Oct 31 03:45:22.080064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.073687 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:45:22.080064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.078216 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa0ef49-ec77-405b-907f-f30d5878209e" path="/var/lib/kubelet/pods/afa0ef49-ec77-405b-907f-f30d5878209e/volumes" Oct 31 03:45:22.080064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.079409 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" path="/var/lib/kubelet/pods/e702e1d3-994f-4787-84ab-e4c991329ef2/volumes" Oct 31 03:45:22.137491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.136396 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": context deadline exceeded" Oct 31 03:45:22.138161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.138129 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.167097 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.167029 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.191245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.167746 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:45:22.192553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.191760 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:45:22.192553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.192270 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.332795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.332666 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.356330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.356290 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.560070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.559119 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.560070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.559489 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:45:22.594430 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.592475 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:22.594430 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.592828 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:45:22.671905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:22.598296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:23.122309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:23.093987 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn"] Oct 31 03:45:23.137814 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:45:23.126478 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec03780_b3bf_4d80_a07e_cce1cc231978.slice/crio-a49934f78f0370446ba9f82caf6805440d4cf29a2a1c7749d48ddeabad9333a9 WatchSource:0}: Error finding container a49934f78f0370446ba9f82caf6805440d4cf29a2a1c7749d48ddeabad9333a9: Status 404 returned error can't find the container with id a49934f78f0370446ba9f82caf6805440d4cf29a2a1c7749d48ddeabad9333a9 Oct 31 03:45:23.241128 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:23.241093 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm"] Oct 31 03:45:23.530111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:23.528857 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" event={"ID":"c463623f-4b59-41e0-a7cf-c5914f027845","Type":"ContainerStarted","Data":"7af52cb9781a4bf1e667b58d803f71d3f90d36fc1dc2291723e484366c34dc3a"} Oct 31 03:45:23.547620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:23.547574 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" event={"ID":"9ec03780-b3bf-4d80-a07e-cce1cc231978","Type":"ContainerStarted","Data":"a49934f78f0370446ba9f82caf6805440d4cf29a2a1c7749d48ddeabad9333a9"} Oct 31 03:45:24.881102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:24.881004 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" event={"ID":"e45b2a27-9d07-4cde-ba44-8409b01e8057","Type":"ContainerStarted","Data":"5b8dd7aff32633356aeaefee5760e87286e9b87bd94b1ef7f25ba4b5b7f5654b"} Oct 31 03:45:24.943565 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:24.943499 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" podStartSLOduration=24.943474159 podStartE2EDuration="24.943474159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:24.905948669 +0000 UTC m=+1749.100471791" watchObservedRunningTime="2026-10-31 03:45:24.943474159 +0000 UTC m=+1749.137997280" Oct 31 03:45:24.953495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:24.953459 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" event={"ID":"9ec03780-b3bf-4d80-a07e-cce1cc231978","Type":"ContainerStarted","Data":"ff37fed470cc4cb1b62ec5414551ee4b8e6a4e0b27c51f07651e2f2bf93d1af5"} Oct 31 03:45:24.953795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:24.953771 5520 scope.go:122] "RemoveContainer" containerID="ff37fed470cc4cb1b62ec5414551ee4b8e6a4e0b27c51f07651e2f2bf93d1af5" Oct 31 03:45:24.956685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:24.956659 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" event={"ID":"c463623f-4b59-41e0-a7cf-c5914f027845","Type":"ContainerStarted","Data":"1f0c987e4e153c454b43f608ed498093935d16dac383a370714f1623e88b3ad9"} Oct 31 03:45:24.956992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:24.956974 5520 scope.go:122] "RemoveContainer" containerID="1f0c987e4e153c454b43f608ed498093935d16dac383a370714f1623e88b3ad9" Oct 31 03:45:26.035240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.035183 5520 generic.go:356] "Generic (PLEG): container finished" podID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerID="5b8dd7aff32633356aeaefee5760e87286e9b87bd94b1ef7f25ba4b5b7f5654b" exitCode=60 Oct 31 03:45:26.035729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.035492 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" event={"ID":"e45b2a27-9d07-4cde-ba44-8409b01e8057","Type":"ContainerDied","Data":"5b8dd7aff32633356aeaefee5760e87286e9b87bd94b1ef7f25ba4b5b7f5654b"} Oct 31 03:45:26.035729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.035556 5520 scope.go:122] "RemoveContainer" containerID="a78cd431f20b59751173fd913026aa59c49bd4a6db9c1753a7e3370b18633c31" Oct 31 03:45:26.035860 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.035834 5520 scope.go:122] "RemoveContainer" containerID="5b8dd7aff32633356aeaefee5760e87286e9b87bd94b1ef7f25ba4b5b7f5654b" Oct 31 03:45:26.036414 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:26.036385 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x_odh-ai-gateway-infra(e45b2a27-9d07-4cde-ba44-8409b01e8057)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" Oct 31 03:45:26.038942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.038917 5520 generic.go:356] "Generic (PLEG): container finished" podID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerID="ff37fed470cc4cb1b62ec5414551ee4b8e6a4e0b27c51f07651e2f2bf93d1af5" exitCode=60 Oct 31 03:45:26.041120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.039447 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" event={"ID":"9ec03780-b3bf-4d80-a07e-cce1cc231978","Type":"ContainerDied","Data":"ff37fed470cc4cb1b62ec5414551ee4b8e6a4e0b27c51f07651e2f2bf93d1af5"} Oct 31 03:45:26.041372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.041353 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" event={"ID":"9ec03780-b3bf-4d80-a07e-cce1cc231978","Type":"ContainerStarted","Data":"617d2eb7c794a70540beb2884db2f9b36cc8ff05db8b0a838e7b2adb064e79d6"} Oct 31 03:45:26.044183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.044161 5520 generic.go:356] "Generic (PLEG): container finished" podID="c463623f-4b59-41e0-a7cf-c5914f027845" containerID="1f0c987e4e153c454b43f608ed498093935d16dac383a370714f1623e88b3ad9" exitCode=60 Oct 31 03:45:26.044458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:26.044442 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" event={"ID":"c463623f-4b59-41e0-a7cf-c5914f027845","Type":"ContainerDied","Data":"1f0c987e4e153c454b43f608ed498093935d16dac383a370714f1623e88b3ad9"} Oct 31 03:45:27.059593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.059557 5520 scope.go:122] "RemoveContainer" containerID="5b8dd7aff32633356aeaefee5760e87286e9b87bd94b1ef7f25ba4b5b7f5654b" Oct 31 03:45:27.060310 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:27.060131 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x_odh-ai-gateway-infra(e45b2a27-9d07-4cde-ba44-8409b01e8057)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" Oct 31 03:45:27.062059 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.062023 5520 generic.go:356] "Generic (PLEG): container finished" podID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerID="617d2eb7c794a70540beb2884db2f9b36cc8ff05db8b0a838e7b2adb064e79d6" exitCode=60 Oct 31 03:45:27.062186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.062159 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" event={"ID":"9ec03780-b3bf-4d80-a07e-cce1cc231978","Type":"ContainerDied","Data":"617d2eb7c794a70540beb2884db2f9b36cc8ff05db8b0a838e7b2adb064e79d6"} Oct 31 03:45:27.062306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.062250 5520 scope.go:122] "RemoveContainer" containerID="ff37fed470cc4cb1b62ec5414551ee4b8e6a4e0b27c51f07651e2f2bf93d1af5" Oct 31 03:45:27.063110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.062636 5520 scope.go:122] "RemoveContainer" containerID="617d2eb7c794a70540beb2884db2f9b36cc8ff05db8b0a838e7b2adb064e79d6" Oct 31 03:45:27.063257 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:27.063125 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn_odh-ai-gateway-infra(9ec03780-b3bf-4d80-a07e-cce1cc231978)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" Oct 31 03:45:27.073234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.067357 5520 generic.go:356] "Generic (PLEG): container finished" podID="c463623f-4b59-41e0-a7cf-c5914f027845" containerID="4d49e55d2de7e6a786e8ebf7a610415b1454a58057619671d6c940551c92bab0" exitCode=60 Oct 31 03:45:27.073234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.067396 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" event={"ID":"c463623f-4b59-41e0-a7cf-c5914f027845","Type":"ContainerDied","Data":"4d49e55d2de7e6a786e8ebf7a610415b1454a58057619671d6c940551c92bab0"} Oct 31 03:45:27.073234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.068079 5520 scope.go:122] "RemoveContainer" containerID="4d49e55d2de7e6a786e8ebf7a610415b1454a58057619671d6c940551c92bab0" Oct 31 03:45:27.073234 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:27.068603 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm_odh-ai-gateway-infra(c463623f-4b59-41e0-a7cf-c5914f027845)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" Oct 31 03:45:27.106302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.101311 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" podStartSLOduration=27.101287205 podStartE2EDuration="27.101287205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:26.103948188 +0000 UTC m=+1750.298471340" watchObservedRunningTime="2026-10-31 03:45:27.101287205 +0000 UTC m=+1751.295810334" Oct 31 03:45:27.117162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:27.117132 5520 scope.go:122] "RemoveContainer" containerID="1f0c987e4e153c454b43f608ed498093935d16dac383a370714f1623e88b3ad9" Oct 31 03:45:28.084307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:28.084274 5520 scope.go:122] "RemoveContainer" containerID="617d2eb7c794a70540beb2884db2f9b36cc8ff05db8b0a838e7b2adb064e79d6" Oct 31 03:45:28.084844 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:28.084815 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn_odh-ai-gateway-infra(9ec03780-b3bf-4d80-a07e-cce1cc231978)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" Oct 31 03:45:28.086780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:28.086756 5520 scope.go:122] "RemoveContainer" containerID="4d49e55d2de7e6a786e8ebf7a610415b1454a58057619671d6c940551c92bab0" Oct 31 03:45:28.087291 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:28.087262 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm_odh-ai-gateway-infra(c463623f-4b59-41e0-a7cf-c5914f027845)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" Oct 31 03:45:30.831358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:30.831307 5520 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:45:30.831954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:30.831374 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:45:33.162856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:33.162820 5520 generic.go:356] "Generic (PLEG): container finished" podID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerID="03e588934e172c2c07286643fbcf4ea37bc5984af5978292ad545b1397442455" exitCode=0 Oct 31 03:45:33.163375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:33.162897 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" event={"ID":"33e15ae6-cdb8-46aa-b7b8-983b7c19920c","Type":"ContainerDied","Data":"03e588934e172c2c07286643fbcf4ea37bc5984af5978292ad545b1397442455"} Oct 31 03:45:33.163454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:33.163440 5520 scope.go:122] "RemoveContainer" containerID="03e588934e172c2c07286643fbcf4ea37bc5984af5978292ad545b1397442455" Oct 31 03:45:33.165900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:33.165871 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:45:33.166042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:33.165963 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:45:33.166042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:33.166014 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:45:34.207715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:34.207674 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" event={"ID":"33e15ae6-cdb8-46aa-b7b8-983b7c19920c","Type":"ContainerStarted","Data":"0c893f22fa4710f8b4c26fbf783b6c2a992702086104c30e2e30fdd64d28a0f0"} Oct 31 03:45:35.025000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:35.024963 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:45:35.025349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:35.025322 5520 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:45:35.025436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:35.025388 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:45:36.059284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:36.049900 5520 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:45:36.059284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:36.049994 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:45:36.914236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:36.913853 5520 scope.go:122] "RemoveContainer" containerID="5b8dd7aff32633356aeaefee5760e87286e9b87bd94b1ef7f25ba4b5b7f5654b" Oct 31 03:45:36.936277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:36.914975 5520 scope.go:122] "RemoveContainer" containerID="4d49e55d2de7e6a786e8ebf7a610415b1454a58057619671d6c940551c92bab0" Oct 31 03:45:37.024985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:37.024950 5520 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:45:37.025151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:37.024999 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:45:38.024523 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.024478 5520 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:45:38.025021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.024537 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:45:38.419236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.415753 5520 generic.go:356] "Generic (PLEG): container finished" podID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerID="ad94365f4eb38497b54b3c1ab0bfe53ca9be59834f28fa4634c0399f57c30d8b" exitCode=60 Oct 31 03:45:38.419236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.416059 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" event={"ID":"e45b2a27-9d07-4cde-ba44-8409b01e8057","Type":"ContainerDied","Data":"ad94365f4eb38497b54b3c1ab0bfe53ca9be59834f28fa4634c0399f57c30d8b"} Oct 31 03:45:38.419236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.416124 5520 scope.go:122] "RemoveContainer" containerID="5b8dd7aff32633356aeaefee5760e87286e9b87bd94b1ef7f25ba4b5b7f5654b" Oct 31 03:45:38.419236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.416752 5520 scope.go:122] "RemoveContainer" containerID="ad94365f4eb38497b54b3c1ab0bfe53ca9be59834f28fa4634c0399f57c30d8b" Oct 31 03:45:38.419236 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:38.417291 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x_odh-ai-gateway-infra(e45b2a27-9d07-4cde-ba44-8409b01e8057)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" Oct 31 03:45:38.457619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.457579 5520 generic.go:356] "Generic (PLEG): container finished" podID="c463623f-4b59-41e0-a7cf-c5914f027845" containerID="76085963357e79aa90016668ffaed56029fd3b90fab0c1c345989050839d02f8" exitCode=60 Oct 31 03:45:38.457844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.457629 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" event={"ID":"c463623f-4b59-41e0-a7cf-c5914f027845","Type":"ContainerDied","Data":"76085963357e79aa90016668ffaed56029fd3b90fab0c1c345989050839d02f8"} Oct 31 03:45:38.458256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.458187 5520 scope.go:122] "RemoveContainer" containerID="76085963357e79aa90016668ffaed56029fd3b90fab0c1c345989050839d02f8" Oct 31 03:45:38.458721 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:38.458694 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm_odh-ai-gateway-infra(c463623f-4b59-41e0-a7cf-c5914f027845)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" Oct 31 03:45:38.461663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.461643 5520 scope.go:122] "RemoveContainer" containerID="4d49e55d2de7e6a786e8ebf7a610415b1454a58057619671d6c940551c92bab0" Oct 31 03:45:38.992238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:38.958434 5520 scope.go:122] "RemoveContainer" containerID="617d2eb7c794a70540beb2884db2f9b36cc8ff05db8b0a838e7b2adb064e79d6" Oct 31 03:45:39.042182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.041075 5520 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 31 03:45:39.042182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.041128 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 31 03:45:39.362580 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:39.362485 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:45:39.456236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.456188 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x"] Oct 31 03:45:39.495925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.495896 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm"] Oct 31 03:45:39.864026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.863168 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx"] Oct 31 03:45:39.864244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.864161 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:39.864244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.864179 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:39.864473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.864458 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e702e1d3-994f-4787-84ab-e4c991329ef2" containerName="cleanup" Oct 31 03:45:39.887245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.870097 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:39.898765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.891693 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b68780\"/\"kube-root-ca.crt\"" Oct 31 03:45:39.898765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.892494 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b68780\"/\"test-model-b68780-kserve-self-signed-certs\"" Oct 31 03:45:39.898765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.892633 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b68780\"/\"default-dockercfg-cbcf4\"" Oct 31 03:45:39.898765 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.893644 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b68780\"/\"openshift-service-ca.crt\"" Oct 31 03:45:39.906400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.906356 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx"] Oct 31 03:45:39.978959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.978922 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfzn\" (UniqueName: \"kubernetes.io/projected/9e552324-bc28-429a-afa7-f882d6356760-kube-api-access-nmfzn\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:39.979279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.979254 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-home\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:39.979446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.979427 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e552324-bc28-429a-afa7-f882d6356760-tls-certs\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:39.979640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.979621 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-model-cache\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:39.980079 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.980058 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-tmp-dir\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:39.980287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:39.980267 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-dshm\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.013295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.013259 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:40.018606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.018580 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:40.029829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.029806 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:45:40.030529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.030505 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:45:40.031280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.031265 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:45:40.056257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.056225 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw"] Oct 31 03:45:40.058306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058277 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.058495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058479 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.058613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058601 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.058708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058696 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.058807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058797 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.058901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058880 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.059015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058931 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.059015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058939 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.059015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058955 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.059015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.058962 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.059287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.059271 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.059287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.059293 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.059393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.059303 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.059393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.059316 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.059393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.059326 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" containerName="cleanup" Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.173398 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c463623f-4b59-41e0-a7cf-c5914f027845-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463623f-4b59-41e0-a7cf-c5914f027845-maas-api-service-ca\") pod \"c463623f-4b59-41e0-a7cf-c5914f027845\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.173605 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45b2a27-9d07-4cde-ba44-8409b01e8057-kube-api-access-qmpc2\" (UniqueName: \"kubernetes.io/projected/e45b2a27-9d07-4cde-ba44-8409b01e8057-kube-api-access-qmpc2\") pod \"e45b2a27-9d07-4cde-ba44-8409b01e8057\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.173705 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c463623f-4b59-41e0-a7cf-c5914f027845-kube-api-access-42lbg\" (UniqueName: \"kubernetes.io/projected/c463623f-4b59-41e0-a7cf-c5914f027845-kube-api-access-42lbg\") pod \"c463623f-4b59-41e0-a7cf-c5914f027845\" (UID: \"c463623f-4b59-41e0-a7cf-c5914f027845\") " Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.173873 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45b2a27-9d07-4cde-ba44-8409b01e8057-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45b2a27-9d07-4cde-ba44-8409b01e8057-maas-api-service-ca\") pod \"e45b2a27-9d07-4cde-ba44-8409b01e8057\" (UID: \"e45b2a27-9d07-4cde-ba44-8409b01e8057\") " Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.174181 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-tmp-dir\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.174295 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-dshm\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.174501 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfzn\" (UniqueName: \"kubernetes.io/projected/9e552324-bc28-429a-afa7-f882d6356760-kube-api-access-nmfzn\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.174558 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-home\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.174589 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e552324-bc28-429a-afa7-f882d6356760-tls-certs\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.185297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.174687 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-model-cache\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.187253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.187217 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c463623f-4b59-41e0-a7cf-c5914f027845-maas-api-service-ca" pod "c463623f-4b59-41e0-a7cf-c5914f027845" (UID: "c463623f-4b59-41e0-a7cf-c5914f027845"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:40.235445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.232933 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-tmp-dir\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.235445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.233306 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-model-cache\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.235445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.233960 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-home\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.265257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.236436 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45b2a27-9d07-4cde-ba44-8409b01e8057-maas-api-service-ca" pod "e45b2a27-9d07-4cde-ba44-8409b01e8057" (UID: "e45b2a27-9d07-4cde-ba44-8409b01e8057"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:40.265257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.254416 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw"] Oct 31 03:45:40.265257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.254534 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.494276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.481997 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e45b2a27-9d07-4cde-ba44-8409b01e8057-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:40.494276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.482041 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c463623f-4b59-41e0-a7cf-c5914f027845-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:40.494276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.482583 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45b2a27-9d07-4cde-ba44-8409b01e8057-kube-api-access-qmpc2" pod "e45b2a27-9d07-4cde-ba44-8409b01e8057" (UID: "e45b2a27-9d07-4cde-ba44-8409b01e8057"). InnerVolumeSpecName "kube-api-access-qmpc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:40.494276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.493912 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c463623f-4b59-41e0-a7cf-c5914f027845-kube-api-access-42lbg" pod "c463623f-4b59-41e0-a7cf-c5914f027845" (UID: "c463623f-4b59-41e0-a7cf-c5914f027845"). InnerVolumeSpecName "kube-api-access-42lbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:40.502236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.500041 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfzn\" (UniqueName: \"kubernetes.io/projected/9e552324-bc28-429a-afa7-f882d6356760-kube-api-access-nmfzn\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.506995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.505648 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-dshm\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.508998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.508905 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e552324-bc28-429a-afa7-f882d6356760-tls-certs\") pod \"test-model-b68780-kserve-55b76b45cc-569bx\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.614761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.614677 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee39d7-1250-4fdd-94a3-bf845aafe070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.615143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.615116 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwjs\" (UniqueName: \"kubernetes.io/projected/9cee39d7-1250-4fdd-94a3-bf845aafe070-kube-api-access-lbwjs\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.615976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.615956 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lbg\" (UniqueName: \"kubernetes.io/projected/c463623f-4b59-41e0-a7cf-c5914f027845-kube-api-access-42lbg\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:40.616178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.616158 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpc2\" (UniqueName: \"kubernetes.io/projected/e45b2a27-9d07-4cde-ba44-8409b01e8057-kube-api-access-qmpc2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:40.688380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.688345 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:40.777071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.776963 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee39d7-1250-4fdd-94a3-bf845aafe070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.777275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.777079 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwjs\" (UniqueName: \"kubernetes.io/projected/9cee39d7-1250-4fdd-94a3-bf845aafe070-kube-api-access-lbwjs\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.780474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.780431 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee39d7-1250-4fdd-94a3-bf845aafe070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.803463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.803434 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw"] Oct 31 03:45:40.805858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.805823 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.806045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.806025 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.806878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.806864 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" containerName="cleanup" Oct 31 03:45:40.808443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.808418 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwjs\" (UniqueName: \"kubernetes.io/projected/9cee39d7-1250-4fdd-94a3-bf845aafe070-kube-api-access-lbwjs\") pod \"maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.814004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.813926 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" Oct 31 03:45:40.814632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.814598 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:40.819399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.817943 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm" event={"ID":"c463623f-4b59-41e0-a7cf-c5914f027845","Type":"ContainerDied","Data":"7af52cb9781a4bf1e667b58d803f71d3f90d36fc1dc2291723e484366c34dc3a"} Oct 31 03:45:40.819399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.818013 5520 scope.go:122] "RemoveContainer" containerID="76085963357e79aa90016668ffaed56029fd3b90fab0c1c345989050839d02f8" Oct 31 03:45:40.819399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.818143 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:40.821253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.820036 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" Oct 31 03:45:40.821253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.820175 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x" event={"ID":"e45b2a27-9d07-4cde-ba44-8409b01e8057","Type":"ContainerDied","Data":"a55ff9072365ff89d02667e18ac211d0fdc33920dd2c5793be004af0b9b02519"} Oct 31 03:45:40.829648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.829596 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw"] Oct 31 03:45:40.832400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.832117 5520 generic.go:356] "Generic (PLEG): container finished" podID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerID="b029624510ce81c89847ff1276ba665784b117ea390180a6979d906052982464" exitCode=60 Oct 31 03:45:40.832795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.832754 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" event={"ID":"9ec03780-b3bf-4d80-a07e-cce1cc231978","Type":"ContainerDied","Data":"b029624510ce81c89847ff1276ba665784b117ea390180a6979d906052982464"} Oct 31 03:45:40.835508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.834559 5520 scope.go:122] "RemoveContainer" containerID="b029624510ce81c89847ff1276ba665784b117ea390180a6979d906052982464" Oct 31 03:45:40.835508 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:40.834959 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn_odh-ai-gateway-infra(9ec03780-b3bf-4d80-a07e-cce1cc231978)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" Oct 31 03:45:40.857027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.857000 5520 scope.go:122] "RemoveContainer" containerID="ad94365f4eb38497b54b3c1ab0bfe53ca9be59834f28fa4634c0399f57c30d8b" Oct 31 03:45:40.899114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.881154 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm"] Oct 31 03:45:40.899114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.889183 5520 scope.go:122] "RemoveContainer" containerID="617d2eb7c794a70540beb2884db2f9b36cc8ff05db8b0a838e7b2adb064e79d6" Oct 31 03:45:40.899114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.890977 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890305-97gzm"] Oct 31 03:45:40.915645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.915585 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x"] Oct 31 03:45:40.940992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.940934 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-46ebfe-29890305-p6w7x"] Oct 31 03:45:40.992039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.992005 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a385d0-48da-4d47-a024-63fdfeeee16f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-zrzqw\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:40.992495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:40.992470 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh2b\" (UniqueName: \"kubernetes.io/projected/23a385d0-48da-4d47-a024-63fdfeeee16f-kube-api-access-zmh2b\") pod \"maas-api-key-cleanup-29890305-zrzqw\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:41.028868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.028603 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn"] Oct 31 03:45:41.095313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.095274 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a385d0-48da-4d47-a024-63fdfeeee16f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-zrzqw\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:41.095830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.095367 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh2b\" (UniqueName: \"kubernetes.io/projected/23a385d0-48da-4d47-a024-63fdfeeee16f-kube-api-access-zmh2b\") pod \"maas-api-key-cleanup-29890305-zrzqw\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:41.096047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.096025 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a385d0-48da-4d47-a024-63fdfeeee16f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-zrzqw\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:41.105341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.105303 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh2b\" (UniqueName: \"kubernetes.io/projected/23a385d0-48da-4d47-a024-63fdfeeee16f-kube-api-access-zmh2b\") pod \"maas-api-key-cleanup-29890305-zrzqw\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:41.134685 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:45:41.134648 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e552324_bc28_429a_afa7_f882d6356760.slice/crio-2f89b709a863f8fc68dbb3b6ef2f176770895ffa504617f5059c408c0817eed7 WatchSource:0}: Error finding container 2f89b709a863f8fc68dbb3b6ef2f176770895ffa504617f5059c408c0817eed7: Status 404 returned error can't find the container with id 2f89b709a863f8fc68dbb3b6ef2f176770895ffa504617f5059c408c0817eed7 Oct 31 03:45:41.137714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.137688 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx"] Oct 31 03:45:41.151828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.151802 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:45:41.187599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.186574 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw"] Oct 31 03:45:41.629348 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:45:41.629318 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a385d0_48da_4d47_a024_63fdfeeee16f.slice/crio-49df84945e4bed515e67c9ab2482d6d8552e715a2cfa1779456e5223e7a781c8 WatchSource:0}: Error finding container 49df84945e4bed515e67c9ab2482d6d8552e715a2cfa1779456e5223e7a781c8: Status 404 returned error can't find the container with id 49df84945e4bed515e67c9ab2482d6d8552e715a2cfa1779456e5223e7a781c8 Oct 31 03:45:41.632225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.632180 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw"] Oct 31 03:45:41.842973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.842944 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" event={"ID":"9cee39d7-1250-4fdd-94a3-bf845aafe070","Type":"ContainerStarted","Data":"d39e7b1e7c6d90d7f206e766c2d873b5adc1b4371cc825a7b58966a91397f466"} Oct 31 03:45:41.843174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.842998 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" event={"ID":"9cee39d7-1250-4fdd-94a3-bf845aafe070","Type":"ContainerStarted","Data":"7fac207db4aead15bdd4ba2bb9dcac3cba00f4a7b7a4a88f4299f12c171bfd6b"} Oct 31 03:45:41.844790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.844763 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" event={"ID":"9e552324-bc28-429a-afa7-f882d6356760","Type":"ContainerStarted","Data":"d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73"} Oct 31 03:45:41.844941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.844797 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" event={"ID":"9e552324-bc28-429a-afa7-f882d6356760","Type":"ContainerStarted","Data":"2f89b709a863f8fc68dbb3b6ef2f176770895ffa504617f5059c408c0817eed7"} Oct 31 03:45:41.846081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.846053 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" event={"ID":"23a385d0-48da-4d47-a024-63fdfeeee16f","Type":"ContainerStarted","Data":"49df84945e4bed515e67c9ab2482d6d8552e715a2cfa1779456e5223e7a781c8"} Oct 31 03:45:41.859717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:41.859676 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" podStartSLOduration=41.859663889 podStartE2EDuration="41.859663889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:41.85745536 +0000 UTC m=+1766.051978501" watchObservedRunningTime="2026-10-31 03:45:41.859663889 +0000 UTC m=+1766.054187047" Oct 31 03:45:42.011318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.007766 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podStartSLOduration=3.007744616 podStartE2EDuration="3.007744616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:41.895183265 +0000 UTC m=+1766.089706420" watchObservedRunningTime="2026-10-31 03:45:42.007744616 +0000 UTC m=+1766.202267738" Oct 31 03:45:42.026266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.023468 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c463623f-4b59-41e0-a7cf-c5914f027845" path="/var/lib/kubelet/pods/c463623f-4b59-41e0-a7cf-c5914f027845/volumes" Oct 31 03:45:42.026266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.024357 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45b2a27-9d07-4cde-ba44-8409b01e8057" path="/var/lib/kubelet/pods/e45b2a27-9d07-4cde-ba44-8409b01e8057/volumes" Oct 31 03:45:42.360409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.360379 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:42.400919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.400882 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec03780-b3bf-4d80-a07e-cce1cc231978-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec03780-b3bf-4d80-a07e-cce1cc231978-maas-api-service-ca\") pod \"9ec03780-b3bf-4d80-a07e-cce1cc231978\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " Oct 31 03:45:42.402182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.402156 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec03780-b3bf-4d80-a07e-cce1cc231978-maas-api-service-ca" pod "9ec03780-b3bf-4d80-a07e-cce1cc231978" (UID: "9ec03780-b3bf-4d80-a07e-cce1cc231978"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:42.403782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.402633 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec03780-b3bf-4d80-a07e-cce1cc231978-kube-api-access-sn464\" (UniqueName: \"kubernetes.io/projected/9ec03780-b3bf-4d80-a07e-cce1cc231978-kube-api-access-sn464\") pod \"9ec03780-b3bf-4d80-a07e-cce1cc231978\" (UID: \"9ec03780-b3bf-4d80-a07e-cce1cc231978\") " Oct 31 03:45:42.404296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.403686 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj"] Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.408359 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec03780-b3bf-4d80-a07e-cce1cc231978-kube-api-access-sn464" pod "9ec03780-b3bf-4d80-a07e-cce1cc231978" (UID: "9ec03780-b3bf-4d80-a07e-cce1cc231978"). InnerVolumeSpecName "kube-api-access-sn464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.413026 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn464\" (UniqueName: \"kubernetes.io/projected/9ec03780-b3bf-4d80-a07e-cce1cc231978-kube-api-access-sn464\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.413051 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec03780-b3bf-4d80-a07e-cce1cc231978-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.414450 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.414470 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.414530 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.414538 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.415741 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:42.417868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.415815 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:42.422034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.422011 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.430173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.427542 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj"] Oct 31 03:45:42.514750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.514722 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71306f4d-3435-47a9-9cc6-6271297e8de7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.515273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.515246 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42hx\" (UniqueName: \"kubernetes.io/projected/71306f4d-3435-47a9-9cc6-6271297e8de7-kube-api-access-t42hx\") pod \"maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.625229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.623092 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42hx\" (UniqueName: \"kubernetes.io/projected/71306f4d-3435-47a9-9cc6-6271297e8de7-kube-api-access-t42hx\") pod \"maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.625229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.623217 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71306f4d-3435-47a9-9cc6-6271297e8de7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.625229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.624230 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71306f4d-3435-47a9-9cc6-6271297e8de7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.684658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.684625 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42hx\" (UniqueName: \"kubernetes.io/projected/71306f4d-3435-47a9-9cc6-6271297e8de7-kube-api-access-t42hx\") pod \"maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.695352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.695318 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:42.746425 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.746334 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:45:42.864475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.864444 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" Oct 31 03:45:42.864642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.864463 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn" event={"ID":"9ec03780-b3bf-4d80-a07e-cce1cc231978","Type":"ContainerDied","Data":"a49934f78f0370446ba9f82caf6805440d4cf29a2a1c7749d48ddeabad9333a9"} Oct 31 03:45:42.864642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.864548 5520 scope.go:122] "RemoveContainer" containerID="b029624510ce81c89847ff1276ba665784b117ea390180a6979d906052982464" Oct 31 03:45:42.866644 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.866618 5520 generic.go:356] "Generic (PLEG): container finished" podID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerID="d39e7b1e7c6d90d7f206e766c2d873b5adc1b4371cc825a7b58966a91397f466" exitCode=60 Oct 31 03:45:42.866777 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.866745 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" event={"ID":"9cee39d7-1250-4fdd-94a3-bf845aafe070","Type":"ContainerDied","Data":"d39e7b1e7c6d90d7f206e766c2d873b5adc1b4371cc825a7b58966a91397f466"} Oct 31 03:45:42.867096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.867077 5520 scope.go:122] "RemoveContainer" containerID="d39e7b1e7c6d90d7f206e766c2d873b5adc1b4371cc825a7b58966a91397f466" Oct 31 03:45:42.925504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.925466 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" event={"ID":"23a385d0-48da-4d47-a024-63fdfeeee16f","Type":"ContainerStarted","Data":"f7b3522c0cca81504a9f09817c63081699e08e1fe27fc74791c29ab595256965"} Oct 31 03:45:42.929433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.929402 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn"] Oct 31 03:45:42.936533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.936508 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f28b4-29890305-j6jgn"] Oct 31 03:45:42.944721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:42.944660 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" podStartSLOduration=42.944643527 podStartE2EDuration="42.944643527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:42.942113329 +0000 UTC m=+1767.136636451" watchObservedRunningTime="2026-10-31 03:45:42.944643527 +0000 UTC m=+1767.139166662" Oct 31 03:45:43.047961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.047933 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj"] Oct 31 03:45:43.064381 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:45:43.064351 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71306f4d_3435_47a9_9cc6_6271297e8de7.slice/crio-ecbe23f83df937fbb9809f203389dd81fc7477a38db4e277f4a86401e255bf08 WatchSource:0}: Error finding container ecbe23f83df937fbb9809f203389dd81fc7477a38db4e277f4a86401e255bf08: Status 404 returned error can't find the container with id ecbe23f83df937fbb9809f203389dd81fc7477a38db4e277f4a86401e255bf08 Oct 31 03:45:43.923164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.923091 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" path="/var/lib/kubelet/pods/9ec03780-b3bf-4d80-a07e-cce1cc231978/volumes" Oct 31 03:45:43.936283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.936253 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" event={"ID":"9cee39d7-1250-4fdd-94a3-bf845aafe070","Type":"ContainerStarted","Data":"5a669bed341caff5c448fd490373ca3af5e47e48617198b34bbc85844d235485"} Oct 31 03:45:43.937899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.937874 5520 generic.go:356] "Generic (PLEG): container finished" podID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerID="f7b3522c0cca81504a9f09817c63081699e08e1fe27fc74791c29ab595256965" exitCode=60 Oct 31 03:45:43.938012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.937953 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" event={"ID":"23a385d0-48da-4d47-a024-63fdfeeee16f","Type":"ContainerDied","Data":"f7b3522c0cca81504a9f09817c63081699e08e1fe27fc74791c29ab595256965"} Oct 31 03:45:43.938193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.938174 5520 scope.go:122] "RemoveContainer" containerID="f7b3522c0cca81504a9f09817c63081699e08e1fe27fc74791c29ab595256965" Oct 31 03:45:43.939833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.939803 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" event={"ID":"71306f4d-3435-47a9-9cc6-6271297e8de7","Type":"ContainerStarted","Data":"8b346cb8ec07e2009749b691bd513f2d607d8f7265b31f20a950b8744b9bf836"} Oct 31 03:45:43.939954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.939848 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" event={"ID":"71306f4d-3435-47a9-9cc6-6271297e8de7","Type":"ContainerStarted","Data":"ecbe23f83df937fbb9809f203389dd81fc7477a38db4e277f4a86401e255bf08"} Oct 31 03:45:43.998074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:43.997759 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" podStartSLOduration=43.997738979 podStartE2EDuration="43.997738979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:43.993574697 +0000 UTC m=+1768.188097841" watchObservedRunningTime="2026-10-31 03:45:43.997738979 +0000 UTC m=+1768.192262100" Oct 31 03:45:44.972456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.971924 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" event={"ID":"23a385d0-48da-4d47-a024-63fdfeeee16f","Type":"ContainerStarted","Data":"93f7b102cecbaee9993c785575fa69f6048cec4798d114ea3407c8d0345db42e"} Oct 31 03:45:44.973930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.973900 5520 generic.go:356] "Generic (PLEG): container finished" podID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerID="8b346cb8ec07e2009749b691bd513f2d607d8f7265b31f20a950b8744b9bf836" exitCode=60 Oct 31 03:45:44.974071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.973940 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" event={"ID":"71306f4d-3435-47a9-9cc6-6271297e8de7","Type":"ContainerDied","Data":"8b346cb8ec07e2009749b691bd513f2d607d8f7265b31f20a950b8744b9bf836"} Oct 31 03:45:44.974396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.974371 5520 scope.go:122] "RemoveContainer" containerID="8b346cb8ec07e2009749b691bd513f2d607d8f7265b31f20a950b8744b9bf836" Oct 31 03:45:44.976729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.976630 5520 generic.go:356] "Generic (PLEG): container finished" podID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerID="5a669bed341caff5c448fd490373ca3af5e47e48617198b34bbc85844d235485" exitCode=60 Oct 31 03:45:44.976729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.976710 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" event={"ID":"9cee39d7-1250-4fdd-94a3-bf845aafe070","Type":"ContainerDied","Data":"5a669bed341caff5c448fd490373ca3af5e47e48617198b34bbc85844d235485"} Oct 31 03:45:44.976889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.976769 5520 scope.go:122] "RemoveContainer" containerID="d39e7b1e7c6d90d7f206e766c2d873b5adc1b4371cc825a7b58966a91397f466" Oct 31 03:45:44.977013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:44.976995 5520 scope.go:122] "RemoveContainer" containerID="5a669bed341caff5c448fd490373ca3af5e47e48617198b34bbc85844d235485" Oct 31 03:45:44.977397 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:44.977366 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw_odh-ai-gateway-infra(9cee39d7-1250-4fdd-94a3-bf845aafe070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" Oct 31 03:45:45.989646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:45.989605 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" event={"ID":"71306f4d-3435-47a9-9cc6-6271297e8de7","Type":"ContainerStarted","Data":"9651414d820861d397ad864ba32bae8b2a7548e4b909cbf1fb1296a21f6da554"} Oct 31 03:45:45.992239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:45.992184 5520 scope.go:122] "RemoveContainer" containerID="5a669bed341caff5c448fd490373ca3af5e47e48617198b34bbc85844d235485" Oct 31 03:45:45.992691 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:45.992666 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw_odh-ai-gateway-infra(9cee39d7-1250-4fdd-94a3-bf845aafe070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" Oct 31 03:45:45.994029 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:45.994003 5520 generic.go:356] "Generic (PLEG): container finished" podID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerID="93f7b102cecbaee9993c785575fa69f6048cec4798d114ea3407c8d0345db42e" exitCode=60 Oct 31 03:45:45.994157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:45.994087 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" event={"ID":"23a385d0-48da-4d47-a024-63fdfeeee16f","Type":"ContainerDied","Data":"93f7b102cecbaee9993c785575fa69f6048cec4798d114ea3407c8d0345db42e"} Oct 31 03:45:45.994157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:45.994149 5520 scope.go:122] "RemoveContainer" containerID="f7b3522c0cca81504a9f09817c63081699e08e1fe27fc74791c29ab595256965" Oct 31 03:45:45.994374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:45.994358 5520 scope.go:122] "RemoveContainer" containerID="93f7b102cecbaee9993c785575fa69f6048cec4798d114ea3407c8d0345db42e" Oct 31 03:45:45.994825 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:45.994737 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-zrzqw_odh-ai-gateway-infra(23a385d0-48da-4d47-a024-63fdfeeee16f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" Oct 31 03:45:47.021164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:47.021121 5520 generic.go:356] "Generic (PLEG): container finished" podID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerID="9651414d820861d397ad864ba32bae8b2a7548e4b909cbf1fb1296a21f6da554" exitCode=60 Oct 31 03:45:47.021870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:47.021477 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" event={"ID":"71306f4d-3435-47a9-9cc6-6271297e8de7","Type":"ContainerDied","Data":"9651414d820861d397ad864ba32bae8b2a7548e4b909cbf1fb1296a21f6da554"} Oct 31 03:45:47.021870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:47.021538 5520 scope.go:122] "RemoveContainer" containerID="8b346cb8ec07e2009749b691bd513f2d607d8f7265b31f20a950b8744b9bf836" Oct 31 03:45:47.021870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:47.021712 5520 scope.go:122] "RemoveContainer" containerID="9651414d820861d397ad864ba32bae8b2a7548e4b909cbf1fb1296a21f6da554" Oct 31 03:45:47.022570 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:47.022525 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj_odh-ai-gateway-infra(71306f4d-3435-47a9-9cc6-6271297e8de7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" Oct 31 03:45:47.025761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:47.025719 5520 scope.go:122] "RemoveContainer" containerID="93f7b102cecbaee9993c785575fa69f6048cec4798d114ea3407c8d0345db42e" Oct 31 03:45:47.026412 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:47.026385 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-zrzqw_odh-ai-gateway-infra(23a385d0-48da-4d47-a024-63fdfeeee16f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" Oct 31 03:45:48.036252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:48.036220 5520 scope.go:122] "RemoveContainer" containerID="9651414d820861d397ad864ba32bae8b2a7548e4b909cbf1fb1296a21f6da554" Oct 31 03:45:48.036656 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:48.036633 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj_odh-ai-gateway-infra(71306f4d-3435-47a9-9cc6-6271297e8de7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" Oct 31 03:45:50.690995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:50.690958 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:50.704153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:50.704124 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:51.128305 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:51.127916 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:45:56.909545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:56.909514 5520 scope.go:122] "RemoveContainer" containerID="5a669bed341caff5c448fd490373ca3af5e47e48617198b34bbc85844d235485" Oct 31 03:45:58.186164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:58.186130 5520 generic.go:356] "Generic (PLEG): container finished" podID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerID="98c42cbded0c89d4f730509d6152d076fd0a904da09eec2a559dedad147e0bda" exitCode=60 Oct 31 03:45:58.186723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:58.186221 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" event={"ID":"9cee39d7-1250-4fdd-94a3-bf845aafe070","Type":"ContainerDied","Data":"98c42cbded0c89d4f730509d6152d076fd0a904da09eec2a559dedad147e0bda"} Oct 31 03:45:58.186723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:58.186297 5520 scope.go:122] "RemoveContainer" containerID="5a669bed341caff5c448fd490373ca3af5e47e48617198b34bbc85844d235485" Oct 31 03:45:58.187226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:58.187078 5520 scope.go:122] "RemoveContainer" containerID="98c42cbded0c89d4f730509d6152d076fd0a904da09eec2a559dedad147e0bda" Oct 31 03:45:58.187706 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:45:58.187673 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw_odh-ai-gateway-infra(9cee39d7-1250-4fdd-94a3-bf845aafe070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" Oct 31 03:45:59.237036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.236994 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw"] Oct 31 03:45:59.484315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.484290 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:45:59.522303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.520991 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5"] Oct 31 03:45:59.554404 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.554226 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cee39d7-1250-4fdd-94a3-bf845aafe070-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee39d7-1250-4fdd-94a3-bf845aafe070-maas-api-service-ca\") pod \"9cee39d7-1250-4fdd-94a3-bf845aafe070\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " Oct 31 03:45:59.554608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.554517 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cee39d7-1250-4fdd-94a3-bf845aafe070-kube-api-access-lbwjs\" (UniqueName: \"kubernetes.io/projected/9cee39d7-1250-4fdd-94a3-bf845aafe070-kube-api-access-lbwjs\") pod \"9cee39d7-1250-4fdd-94a3-bf845aafe070\" (UID: \"9cee39d7-1250-4fdd-94a3-bf845aafe070\") " Oct 31 03:45:59.556094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.556064 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cee39d7-1250-4fdd-94a3-bf845aafe070-maas-api-service-ca" pod "9cee39d7-1250-4fdd-94a3-bf845aafe070" (UID: "9cee39d7-1250-4fdd-94a3-bf845aafe070"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:45:59.564650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.563491 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:45:59.564650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.563524 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:45:59.564650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.563851 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:45:59.564650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.563993 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:45:59.564650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.564180 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:59.564650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.564191 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:59.567490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.566400 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee39d7-1250-4fdd-94a3-bf845aafe070-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:59.569090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.568071 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:45:59.569090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.568119 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:45:59.569090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.568156 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec03780-b3bf-4d80-a07e-cce1cc231978" containerName="cleanup" Oct 31 03:45:59.612990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.612952 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cee39d7-1250-4fdd-94a3-bf845aafe070-kube-api-access-lbwjs" pod "9cee39d7-1250-4fdd-94a3-bf845aafe070" (UID: "9cee39d7-1250-4fdd-94a3-bf845aafe070"). InnerVolumeSpecName "kube-api-access-lbwjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:45:59.619956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.619929 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5"] Oct 31 03:45:59.620990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.620968 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.625945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.625923 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:45:59.668952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.668916 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwjs\" (UniqueName: \"kubernetes.io/projected/9cee39d7-1250-4fdd-94a3-bf845aafe070-kube-api-access-lbwjs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:45:59.770750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.770709 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4mj\" (UniqueName: \"kubernetes.io/projected/2385df23-2dd4-429f-ae01-47a8cf70977a-kube-api-access-9d4mj\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.770933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.770860 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.771230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.771197 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.878337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.878243 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.879601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.879576 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4mj\" (UniqueName: \"kubernetes.io/projected/2385df23-2dd4-429f-ae01-47a8cf70977a-kube-api-access-9d4mj\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.883064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.883032 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.884025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.879356 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.896959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.896923 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.901387 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.900192 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4mj\" (UniqueName: \"kubernetes.io/projected/2385df23-2dd4-429f-ae01-47a8cf70977a-kube-api-access-9d4mj\") pod \"maas-api-revoke-sub-1a2cd1fc-nccg5\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:45:59.910669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.910643 5520 scope.go:122] "RemoveContainer" containerID="9651414d820861d397ad864ba32bae8b2a7548e4b909cbf1fb1296a21f6da554" Oct 31 03:45:59.910849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.910826 5520 scope.go:122] "RemoveContainer" containerID="93f7b102cecbaee9993c785575fa69f6048cec4798d114ea3407c8d0345db42e" Oct 31 03:45:59.941492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:45:59.941466 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:46:00.116698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.115757 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw"] Oct 31 03:46:00.160498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.160367 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj"] Oct 31 03:46:00.217029 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.217002 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" Oct 31 03:46:00.217251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.217044 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw" event={"ID":"9cee39d7-1250-4fdd-94a3-bf845aafe070","Type":"ContainerDied","Data":"7fac207db4aead15bdd4ba2bb9dcac3cba00f4a7b7a4a88f4299f12c171bfd6b"} Oct 31 03:46:00.217251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.217109 5520 scope.go:122] "RemoveContainer" containerID="98c42cbded0c89d4f730509d6152d076fd0a904da09eec2a559dedad147e0bda" Oct 31 03:46:00.243584 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.243553 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw"] Oct 31 03:46:00.251473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.251443 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-dda5cb-29890305-78tgw"] Oct 31 03:46:00.344900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:00.344873 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5"] Oct 31 03:46:00.371459 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:46:00.371426 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2385df23_2dd4_429f_ae01_47a8cf70977a.slice/crio-2229f5e29affa946c22a8339505cad3d63026b7980610ea66d73c35348db867a WatchSource:0}: Error finding container 2229f5e29affa946c22a8339505cad3d63026b7980610ea66d73c35348db867a: Status 404 returned error can't find the container with id 2229f5e29affa946c22a8339505cad3d63026b7980610ea66d73c35348db867a Oct 31 03:46:01.230496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.230466 5520 generic.go:356] "Generic (PLEG): container finished" podID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerID="0f91205fcd77861e184516e0465c4ae06333e929ea87de51fc86071d34b4ef91" exitCode=60 Oct 31 03:46:01.230673 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.230654 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" event={"ID":"23a385d0-48da-4d47-a024-63fdfeeee16f","Type":"ContainerDied","Data":"0f91205fcd77861e184516e0465c4ae06333e929ea87de51fc86071d34b4ef91"} Oct 31 03:46:01.230736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.230708 5520 scope.go:122] "RemoveContainer" containerID="93f7b102cecbaee9993c785575fa69f6048cec4798d114ea3407c8d0345db42e" Oct 31 03:46:01.233692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.233655 5520 generic.go:356] "Generic (PLEG): container finished" podID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerID="f4e18cfa3afe2d40ae0ad624743957acc24b0fb27ae4924d72625bd863abe5f9" exitCode=60 Oct 31 03:46:01.233788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.233703 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" event={"ID":"71306f4d-3435-47a9-9cc6-6271297e8de7","Type":"ContainerDied","Data":"f4e18cfa3afe2d40ae0ad624743957acc24b0fb27ae4924d72625bd863abe5f9"} Oct 31 03:46:01.235851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.235819 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" event={"ID":"2385df23-2dd4-429f-ae01-47a8cf70977a","Type":"ContainerStarted","Data":"559ebcd0d7b44737ad5dba80fae3519ed547a41872c33419a5d591e6c3222807"} Oct 31 03:46:01.235966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.235877 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" event={"ID":"2385df23-2dd4-429f-ae01-47a8cf70977a","Type":"ContainerStarted","Data":"2229f5e29affa946c22a8339505cad3d63026b7980610ea66d73c35348db867a"} Oct 31 03:46:01.247458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.246877 5520 scope.go:122] "RemoveContainer" containerID="9651414d820861d397ad864ba32bae8b2a7548e4b909cbf1fb1296a21f6da554" Oct 31 03:46:01.264722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.264626 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" podStartSLOduration=13.26457938 podStartE2EDuration="13.26457938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:01.262554153 +0000 UTC m=+1785.457077275" watchObservedRunningTime="2026-10-31 03:46:01.26457938 +0000 UTC m=+1785.459102568" Oct 31 03:46:01.604368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.604332 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:46:01.610489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.610465 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:46:01.637515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.637477 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a385d0-48da-4d47-a024-63fdfeeee16f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a385d0-48da-4d47-a024-63fdfeeee16f-maas-api-service-ca\") pod \"23a385d0-48da-4d47-a024-63fdfeeee16f\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " Oct 31 03:46:01.637753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.637733 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71306f4d-3435-47a9-9cc6-6271297e8de7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71306f4d-3435-47a9-9cc6-6271297e8de7-maas-api-service-ca\") pod \"71306f4d-3435-47a9-9cc6-6271297e8de7\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " Oct 31 03:46:01.638095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.638068 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a385d0-48da-4d47-a024-63fdfeeee16f-kube-api-access-zmh2b\" (UniqueName: \"kubernetes.io/projected/23a385d0-48da-4d47-a024-63fdfeeee16f-kube-api-access-zmh2b\") pod \"23a385d0-48da-4d47-a024-63fdfeeee16f\" (UID: \"23a385d0-48da-4d47-a024-63fdfeeee16f\") " Oct 31 03:46:01.638260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.638124 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71306f4d-3435-47a9-9cc6-6271297e8de7-kube-api-access-t42hx\" (UniqueName: \"kubernetes.io/projected/71306f4d-3435-47a9-9cc6-6271297e8de7-kube-api-access-t42hx\") pod \"71306f4d-3435-47a9-9cc6-6271297e8de7\" (UID: \"71306f4d-3435-47a9-9cc6-6271297e8de7\") " Oct 31 03:46:01.639254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.639185 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71306f4d-3435-47a9-9cc6-6271297e8de7-maas-api-service-ca" pod "71306f4d-3435-47a9-9cc6-6271297e8de7" (UID: "71306f4d-3435-47a9-9cc6-6271297e8de7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:01.639580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.639560 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a385d0-48da-4d47-a024-63fdfeeee16f-maas-api-service-ca" pod "23a385d0-48da-4d47-a024-63fdfeeee16f" (UID: "23a385d0-48da-4d47-a024-63fdfeeee16f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:01.650531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.642854 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a385d0-48da-4d47-a024-63fdfeeee16f-kube-api-access-zmh2b" pod "23a385d0-48da-4d47-a024-63fdfeeee16f" (UID: "23a385d0-48da-4d47-a024-63fdfeeee16f"). InnerVolumeSpecName "kube-api-access-zmh2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:01.650531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.648468 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71306f4d-3435-47a9-9cc6-6271297e8de7-kube-api-access-t42hx" pod "71306f4d-3435-47a9-9cc6-6271297e8de7" (UID: "71306f4d-3435-47a9-9cc6-6271297e8de7"). InnerVolumeSpecName "kube-api-access-t42hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:01.740964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.740867 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmh2b\" (UniqueName: \"kubernetes.io/projected/23a385d0-48da-4d47-a024-63fdfeeee16f-kube-api-access-zmh2b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:01.740964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.740904 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t42hx\" (UniqueName: \"kubernetes.io/projected/71306f4d-3435-47a9-9cc6-6271297e8de7-kube-api-access-t42hx\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:01.740964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.740917 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a385d0-48da-4d47-a024-63fdfeeee16f-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:01.740964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.740931 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71306f4d-3435-47a9-9cc6-6271297e8de7-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:01.921402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:01.921366 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" path="/var/lib/kubelet/pods/9cee39d7-1250-4fdd-94a3-bf845aafe070/volumes" Oct 31 03:46:02.246671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.246645 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" Oct 31 03:46:02.246883 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.246779 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw" event={"ID":"23a385d0-48da-4d47-a024-63fdfeeee16f","Type":"ContainerDied","Data":"49df84945e4bed515e67c9ab2482d6d8552e715a2cfa1779456e5223e7a781c8"} Oct 31 03:46:02.246883 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.246873 5520 scope.go:122] "RemoveContainer" containerID="0f91205fcd77861e184516e0465c4ae06333e929ea87de51fc86071d34b4ef91" Oct 31 03:46:02.249006 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.248983 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" Oct 31 03:46:02.249443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.249185 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj" event={"ID":"71306f4d-3435-47a9-9cc6-6271297e8de7","Type":"ContainerDied","Data":"ecbe23f83df937fbb9809f203389dd81fc7477a38db4e277f4a86401e255bf08"} Oct 31 03:46:02.251873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.251846 5520 generic.go:356] "Generic (PLEG): container finished" podID="2385df23-2dd4-429f-ae01-47a8cf70977a" containerID="559ebcd0d7b44737ad5dba80fae3519ed547a41872c33419a5d591e6c3222807" exitCode=0 Oct 31 03:46:02.252015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.251916 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" event={"ID":"2385df23-2dd4-429f-ae01-47a8cf70977a","Type":"ContainerDied","Data":"559ebcd0d7b44737ad5dba80fae3519ed547a41872c33419a5d591e6c3222807"} Oct 31 03:46:02.264156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.264130 5520 scope.go:122] "RemoveContainer" containerID="f4e18cfa3afe2d40ae0ad624743957acc24b0fb27ae4924d72625bd863abe5f9" Oct 31 03:46:02.290151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.290117 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw"] Oct 31 03:46:02.295807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.295779 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-zrzqw"] Oct 31 03:46:02.303618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.303554 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj"] Oct 31 03:46:02.308313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:02.308288 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-880e04-29890305-qh7sj"] Oct 31 03:46:03.478993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.478964 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:46:03.621719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.621625 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2385df23-2dd4-429f-ae01-47a8cf70977a-kube-api-access-9d4mj\" (UniqueName: \"kubernetes.io/projected/2385df23-2dd4-429f-ae01-47a8cf70977a-kube-api-access-9d4mj\") pod \"2385df23-2dd4-429f-ae01-47a8cf70977a\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " Oct 31 03:46:03.621900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.621786 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-xks-ca\") pod \"2385df23-2dd4-429f-ae01-47a8cf70977a\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " Oct 31 03:46:03.621900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.621825 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-openshift-ca\") pod \"2385df23-2dd4-429f-ae01-47a8cf70977a\" (UID: \"2385df23-2dd4-429f-ae01-47a8cf70977a\") " Oct 31 03:46:03.622283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.622247 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-openshift-ca" pod "2385df23-2dd4-429f-ae01-47a8cf70977a" (UID: "2385df23-2dd4-429f-ae01-47a8cf70977a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:03.623051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.623019 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.624298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.624255 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-xks-ca" pod "2385df23-2dd4-429f-ae01-47a8cf70977a" (UID: "2385df23-2dd4-429f-ae01-47a8cf70977a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:03.624485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.624459 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2385df23-2dd4-429f-ae01-47a8cf70977a-kube-api-access-9d4mj" pod "2385df23-2dd4-429f-ae01-47a8cf70977a" (UID: "2385df23-2dd4-429f-ae01-47a8cf70977a"). InnerVolumeSpecName "kube-api-access-9d4mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:03.725390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.725357 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2385df23-2dd4-429f-ae01-47a8cf70977a-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.725390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.725388 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4mj\" (UniqueName: \"kubernetes.io/projected/2385df23-2dd4-429f-ae01-47a8cf70977a-kube-api-access-9d4mj\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:03.924086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.923994 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" path="/var/lib/kubelet/pods/23a385d0-48da-4d47-a024-63fdfeeee16f/volumes" Oct 31 03:46:03.924830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:03.924809 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" path="/var/lib/kubelet/pods/71306f4d-3435-47a9-9cc6-6271297e8de7/volumes" Oct 31 03:46:04.277750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:04.277707 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" event={"ID":"2385df23-2dd4-429f-ae01-47a8cf70977a","Type":"ContainerDied","Data":"2229f5e29affa946c22a8339505cad3d63026b7980610ea66d73c35348db867a"} Oct 31 03:46:04.277750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:04.277739 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5" Oct 31 03:46:04.278013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:04.277760 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2229f5e29affa946c22a8339505cad3d63026b7980610ea66d73c35348db867a" Oct 31 03:46:09.050334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:09.050290 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b"] Oct 31 03:46:09.510603 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:46:09.510560 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:46:17.551048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:17.550888 5520 scope.go:122] "RemoveContainer" containerID="61932498769f2274086f5f14d46ca8e833228714c44c20fc0e8ecd93b7da8a2c" Oct 31 03:46:20.744956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:20.744915 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k"] Oct 31 03:46:24.101846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.095402 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" containerID="cri-o://f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e" gracePeriod=45 Oct 31 03:46:24.188223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.179937 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": dial tcp 10.134.0.158:8000: connect: connection refused" Oct 31 03:46:24.444704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.444677 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:46:24.521679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.521642 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tls-certs\" (UniqueName: \"kubernetes.io/secret/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tls-certs\") pod \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " Oct 31 03:46:24.521893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.521759 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tmp-dir\") pod \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " Oct 31 03:46:24.521893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.521817 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-dshm\") pod \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " Oct 31 03:46:24.522007 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.521959 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-kube-api-access-wm2ph\" (UniqueName: \"kubernetes.io/projected/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-kube-api-access-wm2ph\") pod \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " Oct 31 03:46:24.522110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.522083 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tmp-dir" pod "79b1b755-4ff4-4e73-99f9-fb3f52fb415b" (UID: "79b1b755-4ff4-4e73-99f9-fb3f52fb415b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:24.522167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.522131 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-home\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-home\") pod \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " Oct 31 03:46:24.522370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.522348 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-model-cache\") pod \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\" (UID: \"79b1b755-4ff4-4e73-99f9-fb3f52fb415b\") " Oct 31 03:46:24.522732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.522704 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-model-cache" pod "79b1b755-4ff4-4e73-99f9-fb3f52fb415b" (UID: "79b1b755-4ff4-4e73-99f9-fb3f52fb415b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:24.522918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.522894 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-home" pod "79b1b755-4ff4-4e73-99f9-fb3f52fb415b" (UID: "79b1b755-4ff4-4e73-99f9-fb3f52fb415b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:24.523480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.523456 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.523572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.523486 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.523572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.523522 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.526138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.526105 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-dshm" pod "79b1b755-4ff4-4e73-99f9-fb3f52fb415b" (UID: "79b1b755-4ff4-4e73-99f9-fb3f52fb415b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:24.526315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.526290 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tls-certs" pod "79b1b755-4ff4-4e73-99f9-fb3f52fb415b" (UID: "79b1b755-4ff4-4e73-99f9-fb3f52fb415b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:24.526428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.526406 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-kube-api-access-wm2ph" pod "79b1b755-4ff4-4e73-99f9-fb3f52fb415b" (UID: "79b1b755-4ff4-4e73-99f9-fb3f52fb415b"). InnerVolumeSpecName "kube-api-access-wm2ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:24.625683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.625650 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.625683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.625677 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.625683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.625688 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2ph\" (UniqueName: \"kubernetes.io/projected/79b1b755-4ff4-4e73-99f9-fb3f52fb415b-kube-api-access-wm2ph\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:24.668581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.668490 5520 generic.go:356] "Generic (PLEG): container finished" podID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerID="f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e" exitCode=0 Oct 31 03:46:24.668581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.668531 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" Oct 31 03:46:24.668792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.668571 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" event={"ID":"79b1b755-4ff4-4e73-99f9-fb3f52fb415b","Type":"ContainerDied","Data":"f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e"} Oct 31 03:46:24.668792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.668620 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b" event={"ID":"79b1b755-4ff4-4e73-99f9-fb3f52fb415b","Type":"ContainerDied","Data":"1fccee48dc9025dfdef127996e74bf070fc460cdfb728123cb5079199d796583"} Oct 31 03:46:24.668792 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.668643 5520 scope.go:122] "RemoveContainer" containerID="f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e" Oct 31 03:46:24.682435 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.682414 5520 scope.go:122] "RemoveContainer" containerID="f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e" Oct 31 03:46:24.682764 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:46:24.682745 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e\": container with ID starting with f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e not found: ID does not exist" containerID="f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e" Oct 31 03:46:24.682830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.682776 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e"} err="failed to get container status \"f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e\": rpc error: code = NotFound desc = could not find container \"f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e\": container with ID starting with f49729932499f55ab86b50d8376ce0402fd563801be86add9f34444c1727ac8e not found: ID does not exist" Oct 31 03:46:24.700069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.700039 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b"] Oct 31 03:46:24.707602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:24.707572 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8385c5/test-model-8385c5-kserve-87c4884d7-gx92b"] Oct 31 03:46:25.937964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:25.937874 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" path="/var/lib/kubelet/pods/79b1b755-4ff4-4e73-99f9-fb3f52fb415b/volumes" Oct 31 03:46:31.315161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:31.315110 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm"] Oct 31 03:46:31.330212 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:31.330169 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m"] Oct 31 03:46:35.976228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:35.965134 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" containerID="cri-o://a6c297918b7b485342a2c3d6bc0a25669f9fa9b1fe86a2332845b167d2421460" gracePeriod=45 Oct 31 03:46:35.989233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:35.982411 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.104244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.090536 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": context deadline exceeded" Oct 31 03:46:36.104244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.099948 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.104244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.100039 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.148810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.126413 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.148810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.127581 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.148810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.127920 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.149448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.149348 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.173801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.173585 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:46:36.203138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.174258 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.287315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.258987 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": read tcp 10.134.0.2:35752->10.134.0.159:8000: read: connection reset by peer" Oct 31 03:46:36.287315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.259159 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:46:36.287315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.282216 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.317245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.316418 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerName="payload-processing" containerID="cri-o://1e0c9f30ab12cfbd70a79b7e474ec087d04fee877b3c86bcf944b2b6f1c358ca" gracePeriod=25 Oct 31 03:46:36.377229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.323090 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.377229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.341287 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerName="payload-pre-processing" containerID="cri-o://30dc7b719af2de123c3763ca1af79037d5d5730a951f5898ca4b555d0177a735" gracePeriod=25 Oct 31 03:46:36.377229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.341537 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:46:36.377229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.357302 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.377229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.358626 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.454225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.453789 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.468229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.455458 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 31 03:46:36.568289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.557045 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.591720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.591628 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.684229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.652372 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.684229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.652835 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:46:36.684229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.653246 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:46:36.751237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.736083 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.821294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.819440 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": context deadline exceeded" Oct 31 03:46:36.962241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.961237 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:46:36.999323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.995713 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.999323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:36.996532 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:46:37.022272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.002024 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:46:37.022634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.022599 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:46:37.097668 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.097562 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.101103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.101075 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:46:37.257234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.182940 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.327442 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.257544 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.335363 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.335430 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.335467 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.257587 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": context deadline exceeded" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.257610 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": context deadline exceeded" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.294051 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.295217 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.295248 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.350329 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.356013 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.356075 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.356114 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.356155 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.356194 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.356250 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.362841 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.364114 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.456814 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.457337 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:46:37.475280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.459564 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": dial tcp 10.134.0.159:8000: connect: connection refused" Oct 31 03:46:37.478592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.476884 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:46:37.478592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.476951 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.575455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.569854 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:46:37.575455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.621703 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.575455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.622043 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:46:37.575455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.622102 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.664365 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.665563 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.665792 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.666022 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.666081 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.666120 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.671047 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.734416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.697412 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:46:37.760222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.760158 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.760831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.760810 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:46:37.971642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.969241 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.997413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:37.987126 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:46:38.028236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.020533 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:38.082924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.080038 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:38.087231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.085863 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 31 03:46:38.101616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.100662 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:38.121422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.121388 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:46:38.351562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.328043 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:38.351562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.327463 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:38.403312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.385706 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:46:38.433774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.432949 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": context deadline exceeded" Oct 31 03:46:38.434701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.434068 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": context deadline exceeded" Oct 31 03:46:38.446061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.445369 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:46:38.447536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.446318 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Oct 31 03:46:38.464816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.447788 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:46:38.523683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.523642 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:46:38.584990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.584950 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": dial tcp 10.134.0.159:8000: connect: connection refused" Oct 31 03:46:38.627553 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.627437 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 31 03:46:38.642220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.642172 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:46:38.657842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.657806 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:38.709572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.709541 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:46:38.757020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.756970 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:38.817590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.817531 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:46:38.855658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.855619 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:46:38.857233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:38.857169 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:46:39.033294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.033251 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:39.169812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.169775 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:39.170338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.055187 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:46:39.170710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.055249 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:39.170861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.120862 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:39.211279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.211238 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:39.215282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.040141 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:46:39.215843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.215813 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": dial tcp 10.134.0.159:8000: connect: connection refused" Oct 31 03:46:39.244228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.243300 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:46:39.244228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.243361 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:46:39.261908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.261568 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:46:39.262563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.262539 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:46:39.265065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.265042 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:46:39.278372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.275290 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:46:39.278372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.275348 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:46:39.278372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.275381 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:46:39.285556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.280492 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:46:39.285556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.280665 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:46:39.285556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.284921 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:46:39.286119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.286105 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:46:39.287248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.287233 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:46:39.287491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.287471 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:46:39.288545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.288527 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:46:39.299102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.299080 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:46:39.322643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.322608 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:46:39.322833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.322676 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:46:39.322833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.322708 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:46:39.322833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.322738 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:46:39.346239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.328943 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:46:39.416991 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:46:39.416947 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:46:39.651432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.651401 5520 generic.go:356] "Generic (PLEG): container finished" podID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerID="a6c297918b7b485342a2c3d6bc0a25669f9fa9b1fe86a2332845b167d2421460" exitCode=0 Oct 31 03:46:39.652244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.651469 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" event={"ID":"759215f2-2e3f-4dbd-9009-166bc4e3819d","Type":"ContainerDied","Data":"a6c297918b7b485342a2c3d6bc0a25669f9fa9b1fe86a2332845b167d2421460"} Oct 31 03:46:39.652407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.652392 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" event={"ID":"759215f2-2e3f-4dbd-9009-166bc4e3819d","Type":"ContainerDied","Data":"e4ddd89fa117af43b464e37a2457d48d832390df2e0761fb292848ef55fe3d9d"} Oct 31 03:46:39.653282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.653264 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4ddd89fa117af43b464e37a2457d48d832390df2e0761fb292848ef55fe3d9d" Oct 31 03:46:39.693703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.693676 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:46:39.804686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.804656 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-home\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-home\") pod \"759215f2-2e3f-4dbd-9009-166bc4e3819d\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " Oct 31 03:46:39.805005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.804978 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-home" pod "759215f2-2e3f-4dbd-9009-166bc4e3819d" (UID: "759215f2-2e3f-4dbd-9009-166bc4e3819d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:39.805153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.805135 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759215f2-2e3f-4dbd-9009-166bc4e3819d-kube-api-access-qwk4t\" (UniqueName: \"kubernetes.io/projected/759215f2-2e3f-4dbd-9009-166bc4e3819d-kube-api-access-qwk4t\") pod \"759215f2-2e3f-4dbd-9009-166bc4e3819d\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " Oct 31 03:46:39.805282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.805237 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/759215f2-2e3f-4dbd-9009-166bc4e3819d-tls-certs\" (UniqueName: \"kubernetes.io/secret/759215f2-2e3f-4dbd-9009-166bc4e3819d-tls-certs\") pod \"759215f2-2e3f-4dbd-9009-166bc4e3819d\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " Oct 31 03:46:39.805359 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.805291 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-model-cache\") pod \"759215f2-2e3f-4dbd-9009-166bc4e3819d\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " Oct 31 03:46:39.805420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.805360 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-tmp-dir\") pod \"759215f2-2e3f-4dbd-9009-166bc4e3819d\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " Oct 31 03:46:39.805499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.805485 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-dshm\") pod \"759215f2-2e3f-4dbd-9009-166bc4e3819d\" (UID: \"759215f2-2e3f-4dbd-9009-166bc4e3819d\") " Oct 31 03:46:39.805838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.805784 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-model-cache" pod "759215f2-2e3f-4dbd-9009-166bc4e3819d" (UID: "759215f2-2e3f-4dbd-9009-166bc4e3819d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:39.806156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.806130 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-tmp-dir" pod "759215f2-2e3f-4dbd-9009-166bc4e3819d" (UID: "759215f2-2e3f-4dbd-9009-166bc4e3819d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:39.806967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.806946 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.808783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.808767 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.808907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.808893 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.809512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.809490 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759215f2-2e3f-4dbd-9009-166bc4e3819d-tls-certs" pod "759215f2-2e3f-4dbd-9009-166bc4e3819d" (UID: "759215f2-2e3f-4dbd-9009-166bc4e3819d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:39.811499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.811474 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-dshm" pod "759215f2-2e3f-4dbd-9009-166bc4e3819d" (UID: "759215f2-2e3f-4dbd-9009-166bc4e3819d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:39.827147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.822376 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759215f2-2e3f-4dbd-9009-166bc4e3819d-kube-api-access-qwk4t" pod "759215f2-2e3f-4dbd-9009-166bc4e3819d" (UID: "759215f2-2e3f-4dbd-9009-166bc4e3819d"). InnerVolumeSpecName "kube-api-access-qwk4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:39.914409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.913161 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwk4t\" (UniqueName: \"kubernetes.io/projected/759215f2-2e3f-4dbd-9009-166bc4e3819d-kube-api-access-qwk4t\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.924232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.922078 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759215f2-2e3f-4dbd-9009-166bc4e3819d-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.924232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:39.922124 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759215f2-2e3f-4dbd-9009-166bc4e3819d-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:40.660872 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:40.660839 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k" Oct 31 03:46:40.685767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:40.685731 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k"] Oct 31 03:46:40.693473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:40.693430 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6f28b4/test-model-6f28b4-kserve-6c486dc6bc-qzq5k"] Oct 31 03:46:41.959583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:41.959547 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" path="/var/lib/kubelet/pods/759215f2-2e3f-4dbd-9009-166bc4e3819d/volumes" Oct 31 03:46:45.949809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:45.949775 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9"] Oct 31 03:46:45.950665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:45.950625 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" containerID="cri-o://692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af" gracePeriod=30 Oct 31 03:46:45.957688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:45.957659 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482"] Oct 31 03:46:45.957918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:45.957890 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" containerID="cri-o://81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81" gracePeriod=30 Oct 31 03:46:45.962326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:45.962307 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x"] Oct 31 03:46:45.962675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:45.962613 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" containerID="cri-o://7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c" gracePeriod=30 Oct 31 03:46:46.399557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:46.399150 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:46:46.399557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:46.399237 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:46:46.644607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:46.644571 5520 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:46:46.644784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:46.644634 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:46:47.001710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:46.998418 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:46:47.008737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:47.005319 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:46:51.438562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.438533 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:46:51.463431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.463304 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-certs\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.463621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.463492 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-podinfo\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.463695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.463637 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-token\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-token\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.463759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.463711 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-socket\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.463825 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.463809 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-envoy\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.466281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.463978 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-data\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.466281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.464033 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-kube-api-access-8qv42\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-kube-api-access-8qv42\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.466281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.464138 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istiod-ca-cert\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.466281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.464193 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-credential-socket\") pod \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\" (UID: \"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3\") " Oct 31 03:46:51.466762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.466734 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-certs" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.475194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.475159 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-podinfo" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:51.486705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.482887 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-token" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.486705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.483140 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-socket" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.486705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.485633 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-envoy" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.486705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.485831 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-data" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.489817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.488186 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-kube-api-access-8qv42" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "kube-api-access-8qv42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.489817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.488525 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istiod-ca-cert" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:51.489817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.488779 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-credential-socket" pod "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" (UID: "4cf55357-a8e7-49ed-87ba-99aa5e09b6d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.511746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.511317 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:46:51.512976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.512952 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts"] Oct 31 03:46:51.515875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.515838 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" Oct 31 03:46:51.516033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516017 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" Oct 31 03:46:51.516151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516140 5520 cpu_manager.go:414] "containerMap: removing container" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.516281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516267 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.516420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516407 5520 cpu_manager.go:414] "containerMap: removing container" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" Oct 31 03:46:51.516510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516499 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" Oct 31 03:46:51.516636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516623 5520 cpu_manager.go:414] "containerMap: removing container" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.516747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516734 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.516851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516840 5520 cpu_manager.go:414] "containerMap: removing container" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.516933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.516922 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.517059 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517046 5520 cpu_manager.go:414] "containerMap: removing container" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" Oct 31 03:46:51.517166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517155 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" Oct 31 03:46:51.517280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517269 5520 cpu_manager.go:414] "containerMap: removing container" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.517374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517363 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.517479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517466 5520 cpu_manager.go:414] "containerMap: removing container" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.517589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517578 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.517678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517668 5520 cpu_manager.go:414] "containerMap: removing container" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.517757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517747 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.517865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517854 5520 cpu_manager.go:414] "containerMap: removing container" podUID="2385df23-2dd4-429f-ae01-47a8cf70977a" containerName="revoke-keys" Oct 31 03:46:51.517976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.517961 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2385df23-2dd4-429f-ae01-47a8cf70977a" containerName="revoke-keys" Oct 31 03:46:51.518087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.518076 5520 cpu_manager.go:414] "containerMap: removing container" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" Oct 31 03:46:51.518171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.518161 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" Oct 31 03:46:51.518313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.518298 5520 cpu_manager.go:414] "containerMap: removing container" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" Oct 31 03:46:51.518424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.518412 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" Oct 31 03:46:51.519460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.519443 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:46:51.519667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.519641 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:46:51.520033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520001 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b1b755-4ff4-4e73-99f9-fb3f52fb415b" containerName="main" Oct 31 03:46:51.520164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520152 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.520313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520299 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.520433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520422 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="759215f2-2e3f-4dbd-9009-166bc4e3819d" containerName="main" Oct 31 03:46:51.520520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520510 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cee39d7-1250-4fdd-94a3-bf845aafe070" containerName="cleanup" Oct 31 03:46:51.520623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520610 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.520720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520711 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a385d0-48da-4d47-a024-63fdfeeee16f" containerName="cleanup" Oct 31 03:46:51.520802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520793 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="2385df23-2dd4-429f-ae01-47a8cf70977a" containerName="revoke-keys" Oct 31 03:46:51.520895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520869 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" Oct 31 03:46:51.520996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.520985 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerName="istio-proxy" Oct 31 03:46:51.521088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.521079 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerName="istio-proxy" Oct 31 03:46:51.521181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.521162 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.526091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.526073 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.543149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.542426 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts"] Oct 31 03:46:51.559494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.559452 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l"] Oct 31 03:46:51.561118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.561041 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="71306f4d-3435-47a9-9cc6-6271297e8de7" containerName="cleanup" Oct 31 03:46:51.566689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.566658 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.585882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.585852 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-envoy\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.586062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.585931 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-credential-socket\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.586062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.585976 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-certs\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.586062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586035 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-token\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-token\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586119 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-socket\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586149 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-kube-api-access-d2q4c\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-kube-api-access-d2q4c\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586326 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-podinfo\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586404 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e461da26-40b4-4633-b9fc-bb23d74c9449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e461da26-40b4-4633-b9fc-bb23d74c9449-istiod-ca-cert\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586436 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-data\") pod \"e461da26-40b4-4633-b9fc-bb23d74c9449\" (UID: \"e461da26-40b4-4633-b9fc-bb23d74c9449\") " Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586845 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586910 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.586949 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587033 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587289 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587332 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587377 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587413 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7n5s\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-kube-api-access-g7n5s\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587440 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587740 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587763 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qv42\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-kube-api-access-8qv42\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587780 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587795 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587810 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587824 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587838 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587852 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.587867 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.588592 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-socket" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.589425 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e461da26-40b4-4633-b9fc-bb23d74c9449-istiod-ca-cert" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.589722 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-credential-socket" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.589882 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-certs" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.594652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.592307 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-data" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.605794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.605766 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l"] Oct 31 03:46:51.606741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.606108 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5d9b6-openshift-default-dockercfg-fjkqj\"" Oct 31 03:46:51.610127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.606871 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-envoy" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.610290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.608429 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-podinfo" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:51.610397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.610068 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-kube-api-access-d2q4c" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "kube-api-access-d2q4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.613687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.613653 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-token" pod "e461da26-40b4-4633-b9fc-bb23d74c9449" (UID: "e461da26-40b4-4633-b9fc-bb23d74c9449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.624391 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.624364 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:46:51.660302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.659760 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj"] Oct 31 03:46:51.660817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.660794 5520 cpu_manager.go:414] "containerMap: removing container" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" Oct 31 03:46:51.660929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.660818 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" Oct 31 03:46:51.661088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.661073 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerName="istio-proxy" Oct 31 03:46:51.664989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.664969 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.677372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.674214 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj"] Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.688873 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87478e66-340f-438d-9ab0-0d7784eaa03d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87478e66-340f-438d-9ab0-0d7784eaa03d-istiod-ca-cert\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.688976 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-data\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689036 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-token\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-token\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689146 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-kube-api-access-f2zz5\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-kube-api-access-f2zz5\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689184 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-podinfo\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689352 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-certs\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689440 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-credential-socket\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689488 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-socket\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689534 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-envoy\") pod \"87478e66-340f-438d-9ab0-0d7784eaa03d\" (UID: \"87478e66-340f-438d-9ab0-0d7784eaa03d\") " Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689846 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689892 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689951 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.689983 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690016 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690070 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690127 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690181 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690410 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690451 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690519 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690555 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7n5s\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-kube-api-access-g7n5s\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690579 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690613 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690660 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctww5\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-kube-api-access-ctww5\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690775 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0b94dd-ab27-432b-aa69-5f551939d376-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690826 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.690917 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0b94dd-ab27-432b-aa69-5f551939d376-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691060 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691086 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e461da26-40b4-4633-b9fc-bb23d74c9449-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691104 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691119 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691133 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691148 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691163 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691176 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e461da26-40b4-4633-b9fc-bb23d74c9449-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.691257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.691190 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2q4c\" (UniqueName: \"kubernetes.io/projected/e461da26-40b4-4633-b9fc-bb23d74c9449-kube-api-access-d2q4c\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.694141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.694111 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87478e66-340f-438d-9ab0-0d7784eaa03d-istiod-ca-cert" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:51.694429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.694408 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-data" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.698243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.697170 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-token" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.698243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.697552 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.698243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.697986 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.698243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.698139 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.698699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.698678 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.701448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.701427 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-socket" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.701823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.701803 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-certs" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.702115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.702096 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.702390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.702369 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-credential-socket" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.704102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.704080 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-podinfo" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:51.704280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.704122 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.712937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.712900 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.720227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.715313 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.720227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.709168 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7n5s\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-kube-api-access-g7n5s\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.720227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.715582 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-kube-api-access-f2zz5" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "kube-api-access-f2zz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.727234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.723368 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-envoy" pod "87478e66-340f-438d-9ab0-0d7784eaa03d" (UID: "87478e66-340f-438d-9ab0-0d7784eaa03d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.776655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.776567 5520 generic.go:356] "Generic (PLEG): container finished" podID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" containerID="692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af" exitCode=0 Oct 31 03:46:51.776655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.776619 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" event={"ID":"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3","Type":"ContainerDied","Data":"692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af"} Oct 31 03:46:51.776859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.776706 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" event={"ID":"4cf55357-a8e7-49ed-87ba-99aa5e09b6d3","Type":"ContainerDied","Data":"a945ce27efeecb25355b5abfca8cf64a78156a142416f0cebb1b859159b5fa16"} Oct 31 03:46:51.776859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.776714 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9" Oct 31 03:46:51.776859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.776732 5520 scope.go:122] "RemoveContainer" containerID="692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af" Oct 31 03:46:51.778536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.778512 5520 generic.go:356] "Generic (PLEG): container finished" podID="e461da26-40b4-4633-b9fc-bb23d74c9449" containerID="81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81" exitCode=0 Oct 31 03:46:51.778742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.778588 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" Oct 31 03:46:51.778910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.778602 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" event={"ID":"e461da26-40b4-4633-b9fc-bb23d74c9449","Type":"ContainerDied","Data":"81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81"} Oct 31 03:46:51.779249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.779125 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482" event={"ID":"e461da26-40b4-4633-b9fc-bb23d74c9449","Type":"ContainerDied","Data":"44a092fd35ebf11a773225c60d40458991018371b68405cb8e89f9b2b366852f"} Oct 31 03:46:51.780816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.780774 5520 generic.go:356] "Generic (PLEG): container finished" podID="87478e66-340f-438d-9ab0-0d7784eaa03d" containerID="7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c" exitCode=0 Oct 31 03:46:51.781055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.781032 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" event={"ID":"87478e66-340f-438d-9ab0-0d7784eaa03d","Type":"ContainerDied","Data":"7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c"} Oct 31 03:46:51.781282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.781258 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" event={"ID":"87478e66-340f-438d-9ab0-0d7784eaa03d","Type":"ContainerDied","Data":"1eb62c49c2917345a54330e30cf08a42ecb5e563f7c0b990d258667379d3ffea"} Oct 31 03:46:51.781431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.781357 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x" Oct 31 03:46:51.838298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.838271 5520 scope.go:122] "RemoveContainer" containerID="6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253" Oct 31 03:46:51.862911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.862772 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.862911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.862840 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.862911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.862887 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.863406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863036 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.863406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863081 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.863406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863140 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.863406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863288 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.863406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863345 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9895b9-a592-4acf-a80a-3ef016179bc2-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.863730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863385 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.863730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863539 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.863730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863579 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctww5\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-kube-api-access-ctww5\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.863730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863623 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.863730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863691 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0b94dd-ab27-432b-aa69-5f551939d376-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.864129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863731 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.864129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863835 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0b94dd-ab27-432b-aa69-5f551939d376-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.864129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863871 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9ck\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-kube-api-access-hf9ck\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.864129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.863950 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.864129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.864055 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.864415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.864183 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.867619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.867518 5520 scope.go:122] "RemoveContainer" containerID="692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af" Oct 31 03:46:51.868915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.868886 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.869562 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:46:51.869440 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af\": container with ID starting with 692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af not found: ID does not exist" containerID="692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af" Oct 31 03:46:51.869678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.869577 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af"} err="failed to get container status \"692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af\": rpc error: code = NotFound desc = could not find container \"692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af\": container with ID starting with 692debd578e7ab4d70984e1018dcf035b86bef68d8c4a7d909fdbe3f2a6a61af not found: ID does not exist" Oct 31 03:46:51.869678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.869604 5520 scope.go:122] "RemoveContainer" containerID="6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253" Oct 31 03:46:51.884384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.876582 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.884384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.877061 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0b94dd-ab27-432b-aa69-5f551939d376-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.884384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.883632 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:51.884384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.884059 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.887651 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:46:51.886170 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253\": container with ID starting with 6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253 not found: ID does not exist" containerID="6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253" Oct 31 03:46:51.887651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.886235 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253"} err="failed to get container status \"6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253\": rpc error: code = NotFound desc = could not find container \"6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253\": container with ID starting with 6d43b4794700f0b2475257f383374cff080ea9af9a3cd9374ea2273744ffa253 not found: ID does not exist" Oct 31 03:46:51.887651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.886262 5520 scope.go:122] "RemoveContainer" containerID="81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81" Oct 31 03:46:51.887651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.886426 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.887651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.886685 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889273 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889322 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889341 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87478e66-340f-438d-9ab0-0d7784eaa03d-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889357 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889373 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889388 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zz5\" (UniqueName: \"kubernetes.io/projected/87478e66-340f-438d-9ab0-0d7784eaa03d-kube-api-access-f2zz5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889405 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87478e66-340f-438d-9ab0-0d7784eaa03d-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.889444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.889421 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87478e66-340f-438d-9ab0-0d7784eaa03d-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.903047 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.902981 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0b94dd-ab27-432b-aa69-5f551939d376-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.904632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.904107 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctww5\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-kube-api-access-ctww5\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.917395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.917364 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.927790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.927766 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482"] Oct 31 03:46:51.928216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.928144 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:51.933402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.933347 5520 scope.go:122] "RemoveContainer" containerID="81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81" Oct 31 03:46:51.933715 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:46:51.933692 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81\": container with ID starting with 81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81 not found: ID does not exist" containerID="81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81" Oct 31 03:46:51.933812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.933728 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81"} err="failed to get container status \"81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81\": rpc error: code = NotFound desc = could not find container \"81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81\": container with ID starting with 81c7cebcbeb8958d1b8b7fddcad86d89e024d9468657220449b7096e4885ac81 not found: ID does not exist" Oct 31 03:46:51.933812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.933751 5520 scope.go:122] "RemoveContainer" containerID="7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c" Oct 31 03:46:51.934163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.934145 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-q7482"] Oct 31 03:46:51.940843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.940821 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x"] Oct 31 03:46:51.946194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.946174 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-h459x"] Oct 31 03:46:51.952873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.952851 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9"] Oct 31 03:46:51.956479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.956459 5520 scope.go:122] "RemoveContainer" containerID="7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c" Oct 31 03:46:51.956823 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:46:51.956799 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c\": container with ID starting with 7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c not found: ID does not exist" containerID="7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c" Oct 31 03:46:51.956939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.956833 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c"} err="failed to get container status \"7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c\": rpc error: code = NotFound desc = could not find container \"7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c\": container with ID starting with 7c387ac29a04b6668131d3760da5bea053182a274faca61f9945f808bc7eb99c not found: ID does not exist" Oct 31 03:46:51.959347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.959326 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-7hwp9"] Oct 31 03:46:51.993068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.992858 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.994011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.993449 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9ck\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-kube-api-access-hf9ck\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.994011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.993587 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.994011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.993660 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.994548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.994419 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.994548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.994499 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.999176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.994958 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.999176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.995025 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9895b9-a592-4acf-a80a-3ef016179bc2-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:51.999176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:51.995083 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.002889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.000419 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.002889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.000961 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.002889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.002512 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.002889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.002800 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9895b9-a592-4acf-a80a-3ef016179bc2-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.004168 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.004148 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.005075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.005038 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.007088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.007041 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.010608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.010217 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.018287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.017848 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9ck\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-kube-api-access-hf9ck\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.261640 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:46:52.261604 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acc8610_820f_4fc1_9fc2_760a4db6a08a.slice/crio-709a73ce10e78000bfa6f1f26cc6c973d0aa142a2243d1a1a3968b4da9134f8e WatchSource:0}: Error finding container 709a73ce10e78000bfa6f1f26cc6c973d0aa142a2243d1a1a3968b4da9134f8e: Status 404 returned error can't find the container with id 709a73ce10e78000bfa6f1f26cc6c973d0aa142a2243d1a1a3968b4da9134f8e Oct 31 03:46:52.263409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.263379 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts"] Oct 31 03:46:52.266276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.265722 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.266276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.265802 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.266276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.265844 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.273133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.272707 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l"] Oct 31 03:46:52.290503 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.290464 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.290639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.290561 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.290639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.290610 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.306122 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.305806 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:52.702900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.702864 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj"] Oct 31 03:46:52.730435 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.730381 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.730570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.730477 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.730570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.730525 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:46:52.809264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.809222 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" event={"ID":"1acc8610-820f-4fc1-9fc2-760a4db6a08a","Type":"ContainerStarted","Data":"07ae77079bb80b262c7d4f201d0c4149124c933ce576629c0e5fab757c40174a"} Oct 31 03:46:52.809449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.809290 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" event={"ID":"1acc8610-820f-4fc1-9fc2-760a4db6a08a","Type":"ContainerStarted","Data":"709a73ce10e78000bfa6f1f26cc6c973d0aa142a2243d1a1a3968b4da9134f8e"} Oct 31 03:46:52.813049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.813015 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" event={"ID":"af0b94dd-ab27-432b-aa69-5f551939d376","Type":"ContainerStarted","Data":"3ca3eff0f565482397a354a9e33fc55a750985d57e8e0565184cc4d2a163c3fa"} Oct 31 03:46:52.813217 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.813069 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" event={"ID":"af0b94dd-ab27-432b-aa69-5f551939d376","Type":"ContainerStarted","Data":"b1077e1e346a5a28f28d69b251b665b6c1a72a314dd02ea10b03300e040e9dbe"} Oct 31 03:46:52.818941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.818915 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" event={"ID":"5b9895b9-a592-4acf-a80a-3ef016179bc2","Type":"ContainerStarted","Data":"097653e7b5cd869bae4223b8ad9955d5ea7fe88963244c028b08ff537d892ff2"} Oct 31 03:46:52.837002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.836404 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podStartSLOduration=221.836384625 podStartE2EDuration="3m41.836384625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:52.830859712 +0000 UTC m=+1837.025382833" watchObservedRunningTime="2026-10-31 03:46:52.836384625 +0000 UTC m=+1837.030907747" Oct 31 03:46:52.864427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.864379 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podStartSLOduration=206.864364121 podStartE2EDuration="3m26.864364121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:52.86026739 +0000 UTC m=+1837.054790714" watchObservedRunningTime="2026-10-31 03:46:52.864364121 +0000 UTC m=+1837.058887241" Oct 31 03:46:52.886335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.886263 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:46:52.936240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:52.933361 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:46:53.843036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.842991 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" event={"ID":"5b9895b9-a592-4acf-a80a-3ef016179bc2","Type":"ContainerStarted","Data":"fd5512af9ddd7d0e190a73e8d52c1ea3876ada5d8e1aed7a6e72c3e76cc69f15"} Oct 31 03:46:53.865028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.864970 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podStartSLOduration=207.8649564 podStartE2EDuration="3m27.8649564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:53.864438741 +0000 UTC m=+1838.058961887" watchObservedRunningTime="2026-10-31 03:46:53.8649564 +0000 UTC m=+1838.059479520" Oct 31 03:46:53.916228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.915326 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:46:53.916228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.915395 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:46:53.944230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.940352 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:46:53.944230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.940414 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:46:53.954212 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.954172 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf55357-a8e7-49ed-87ba-99aa5e09b6d3" path="/var/lib/kubelet/pods/4cf55357-a8e7-49ed-87ba-99aa5e09b6d3/volumes" Oct 31 03:46:53.954840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.954822 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87478e66-340f-438d-9ab0-0d7784eaa03d" path="/var/lib/kubelet/pods/87478e66-340f-438d-9ab0-0d7784eaa03d/volumes" Oct 31 03:46:53.956491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:53.956474 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e461da26-40b4-4633-b9fc-bb23d74c9449" path="/var/lib/kubelet/pods/e461da26-40b4-4633-b9fc-bb23d74c9449/volumes" Oct 31 03:46:54.361982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:54.361947 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:46:54.400331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:54.399840 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:46:54.400331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:54.399912 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:46:54.927228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:54.903841 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:46:54.927228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:54.903915 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:46:54.961259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:54.931802 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:46:54.990494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:54.990453 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:46:55.216767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.216713 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.233230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.217397 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:46:55.298636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.298587 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.324855 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.238536 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.325460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.233538 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.325624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.309279 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.325985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.308540 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.354714 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.331286 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.355267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.353345 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.388360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.357615 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:46:55.388360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.371923 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Oct 31 03:46:55.388360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.382707 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:46:55.438352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.438302 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.465339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.465292 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.655325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.654313 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:46:55.667448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.667409 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.675812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.675342 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:55.756449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.756364 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 31 03:46:55.894883 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.894849 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:46:55.895180 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.895152 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:46:55.953227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.947585 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:46:55.953227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:55.947662 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:46:56.125755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.125288 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.142088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.142045 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.354835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.354740 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.355561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.355524 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:46:56.355707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.355595 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:46:56.355848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.355826 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.356194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.355050 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.360903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.356703 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:46:56.383228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.359469 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.385783 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.359498 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:46:56.386325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.386305 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:46:56.386519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.386505 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:46:56.387701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.359517 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": context deadline exceeded" Oct 31 03:46:56.388074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.359535 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": context deadline exceeded" Oct 31 03:46:56.400113 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.399782 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:46:56.470778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.440108 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 31 03:46:56.480283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.471635 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.727248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.529794 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.782376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.664977 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.804535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.690732 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.815726 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.725061 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6"] Oct 31 03:46:56.893283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.892581 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:56.953386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.953349 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:46:56.954992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:56.954715 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115537 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6"] Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115645 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115691 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115746 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115792 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh"] Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115827 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j"] Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115878 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.115927 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:46:57.119273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.117169 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:46:57.142519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.139982 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p75z2\" (UniqueName: \"kubernetes.io/projected/bd63b361-66b8-4cc5-a8db-619eaf42496b-kube-api-access-p75z2\") pod \"payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:46:57.142519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.140184 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd63b361-66b8-4cc5-a8db-619eaf42496b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:46:57.159367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.159321 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:57.160038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.159986 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:46:57.470265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.470222 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": context deadline exceeded" Oct 31 03:46:57.491226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.489502 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:57.491226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.489770 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:46:57.492592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.492558 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p75z2\" (UniqueName: \"kubernetes.io/projected/bd63b361-66b8-4cc5-a8db-619eaf42496b-kube-api-access-p75z2\") pod \"payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:46:57.492951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.492931 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd63b361-66b8-4cc5-a8db-619eaf42496b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.498959 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.499254 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.499304 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.499486 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.499652 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.499819 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": context deadline exceeded" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.500594 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": context deadline exceeded" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.500775 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.500995 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:46:57.516253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.501051 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:46:57.604445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.600798 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": context deadline exceeded" Oct 31 03:46:57.648236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.645595 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:46:57.673230 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:46:57.650586 5520 logging.go:55] [core] [Channel #5783 SubChannel #5784]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.134:50052", ServerName: "10.134.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.134:50052: connect: connection refused" Oct 31 03:46:58.898241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.754151 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:46:58.898241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.897427 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:46:59.050228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.049346 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.063225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.741949 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:46:59.063225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.754978 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd63b361-66b8-4cc5-a8db-619eaf42496b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:46:59.063225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.791154 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:46:59.063225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.915902 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.063225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.926348 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:46:59.063225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.061928 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:46:59.087839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.698912 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b81d65-dockercfg-g4622\"" Oct 31 03:46:59.087839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.698973 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b81d65\"" Oct 31 03:46:59.087839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.789933 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:46:59.087839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.076876 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:46:59.087839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.562292 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.134:50052\" within 1s: context deadline exceeded" Oct 31 03:46:59.087839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.689337 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:46:59.099230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:57.740288 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.104343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.483551 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:46:59.120232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.119187 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.120232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.897274 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p75z2\" (UniqueName: \"kubernetes.io/projected/bd63b361-66b8-4cc5-a8db-619eaf42496b-kube-api-access-p75z2\") pod \"payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:46:59.134301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.258265 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:46:59.158292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.437501 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.229227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.508276 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.249459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.562173 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded" Oct 31 03:46:59.249459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.486022 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:46:59.249459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.229773 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:46:59.249459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.772734 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.423279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.804558 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:46:59.423279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.871427 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.423279 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:46:58.562409 5520 logging.go:55] [core] [Channel #5781 SubChannel #5782]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.135:50052: operation was canceled" Oct 31 03:46:59.423279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:58.864691 5520 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:46:59.423279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.372365 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.439618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.439477 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": context deadline exceeded" Oct 31 03:46:59.879230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.554462 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" podUID="a4b73f30-4c6f-4f53-a679-84694df92392" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:46:59.879507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.556716 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:46:59.879643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.879617 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:46:59.880616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.561905 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:59.880794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.556782 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:46:59.880931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.880910 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:46:59.881049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.642847 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Oct 31 03:46:59.881693 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.620377 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:46:59.882937 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.882910 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:46:59.883120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.883097 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:46:59.912229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.908516 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 31 03:46:59.929304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.620457 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": context deadline exceeded" Oct 31 03:46:59.931155 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.620504 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": context deadline exceeded" Oct 31 03:46:59.956484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.875092 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:46:59.956784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.956756 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:46:59.958857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.958818 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:46:59.959060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.959039 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:46:59.998226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:46:59.967795 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:47:00.012226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.003466 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 31 03:47:00.050363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.050316 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.080230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.063637 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.080230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.063836 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:47:00.080230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.063954 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 31 03:47:00.081333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.081295 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.180321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.161119 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.180321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.161586 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:47:00.180321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.163798 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.180321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.164058 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": context deadline exceeded" Oct 31 03:47:00.246230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.231730 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.246230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.232164 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.529351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.504329 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:47:00.529351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.504393 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:47:00.529351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.513063 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:47:00.547245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.542807 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.614230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.603963 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.666512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.665635 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:47:00.666803 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.666774 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:47:00.837855 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.836777 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 31 03:47:00.861260 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.856624 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:00.880895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.880854 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.037686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.964374 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.047383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.047310 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.048085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.998072 5520 generic.go:356] "Generic (PLEG): container finished" podID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerID="30dc7b719af2de123c3763ca1af79037d5d5730a951f5898ca4b555d0177a735" exitCode=0 Oct 31 03:47:01.061597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.055334 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:47:01.061597 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:01.058588 5520 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.997s" Oct 31 03:47:01.064972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:00.982357 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.087532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.068697 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 31 03:47:01.087532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.068748 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 31 03:47:01.087532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.070737 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.101278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.091470 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 31 03:47:01.101606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.101555 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 31 03:47:01.116372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.116329 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.173231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.130697 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:47:01.198324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.150301 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": context deadline exceeded" Oct 31 03:47:01.198634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.151620 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.198923 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.151654 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:47:01.199157 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.151677 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 31 03:47:01.199434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.151695 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.199568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.162622 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": context deadline exceeded" Oct 31 03:47:01.199754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.184320 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.199946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.184378 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.200144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.195418 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.202384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.202357 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:47:01.233331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.233284 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.252780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.251983 5520 generic.go:356] "Generic (PLEG): container finished" podID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerID="1e0c9f30ab12cfbd70a79b7e474ec087d04fee877b3c86bcf944b2b6f1c358ca" exitCode=0 Oct 31 03:47:01.254889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.254265 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" event={"ID":"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092","Type":"ContainerDied","Data":"30dc7b719af2de123c3763ca1af79037d5d5730a951f5898ca4b555d0177a735"} Oct 31 03:47:01.255185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.255162 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:47:01.255400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.255385 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:47:01.256126 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.255714 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:47:01.256302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.256280 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" event={"ID":"7e1ca905-81e4-439b-ac0b-facf39d3a7db","Type":"ContainerDied","Data":"1e0c9f30ab12cfbd70a79b7e474ec087d04fee877b3c86bcf944b2b6f1c358ca"} Oct 31 03:47:01.256485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.256467 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:47:01.256778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.256762 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:47:01.256931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.256920 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:47:01.257063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257053 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:47:01.257164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257155 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:47:01.257292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257282 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:47:01.257417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257407 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:47:01.257529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257519 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:47:01.257634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257625 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:47:01.257738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257730 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:47:01.257831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257823 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:47:01.257921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.257913 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:47:01.258015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.258007 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:47:01.258110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.258102 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:47:01.258231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.258220 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:47:01.258358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.258348 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:47:01.265842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.265807 5520 scope.go:122] "RemoveContainer" containerID="1e0c9f30ab12cfbd70a79b7e474ec087d04fee877b3c86bcf944b2b6f1c358ca" Oct 31 03:47:01.311319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.302800 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:47:01.312723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.312693 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:47:01.312909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.312886 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:47:01.445324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.441978 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1ca905-81e4-439b-ac0b-facf39d3a7db-kube-api-access-mkmml\" (UniqueName: \"kubernetes.io/projected/7e1ca905-81e4-439b-ac0b-facf39d3a7db-kube-api-access-mkmml\") pod \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " Oct 31 03:47:01.445324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.442269 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1ca905-81e4-439b-ac0b-facf39d3a7db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ca905-81e4-439b-ac0b-facf39d3a7db-plugins-config-volume\") pod \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\" (UID: \"7e1ca905-81e4-439b-ac0b-facf39d3a7db\") " Oct 31 03:47:01.450811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.449267 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1ca905-81e4-439b-ac0b-facf39d3a7db-plugins-config-volume" pod "7e1ca905-81e4-439b-ac0b-facf39d3a7db" (UID: "7e1ca905-81e4-439b-ac0b-facf39d3a7db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:01.465246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.452121 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:47:01.465246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.456433 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:47:01.509946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.466746 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1ca905-81e4-439b-ac0b-facf39d3a7db-kube-api-access-mkmml" pod "7e1ca905-81e4-439b-ac0b-facf39d3a7db" (UID: "7e1ca905-81e4-439b-ac0b-facf39d3a7db"). InnerVolumeSpecName "kube-api-access-mkmml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:01.545386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.522966 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:01.545386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.534507 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:47:01.568237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.546772 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:47:01.667983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.663956 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkmml\" (UniqueName: \"kubernetes.io/projected/7e1ca905-81e4-439b-ac0b-facf39d3a7db-kube-api-access-mkmml\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:01.667983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.663991 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1ca905-81e4-439b-ac0b-facf39d3a7db-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:01.670316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.670286 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:47:01.806241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.804474 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-plugins-config-volume\") pod \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " Oct 31 03:47:01.806241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.805540 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-kube-api-access-94vx7\" (UniqueName: \"kubernetes.io/projected/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-kube-api-access-94vx7\") pod \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\" (UID: \"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092\") " Oct 31 03:47:02.008766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:01.907647 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-plugins-config-volume" pod "7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" (UID: "7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:02.056661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.056619 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.178:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:02.057882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.057850 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:02.149358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.149281 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:02.162133 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.162096 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-kube-api-access-94vx7" pod "7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" (UID: "7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092"). InnerVolumeSpecName "kube-api-access-94vx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:02.202153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.202119 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vx7\" (UniqueName: \"kubernetes.io/projected/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-kube-api-access-94vx7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:02.202436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.202417 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:02.221682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.221371 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" podUID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" containerName="payload-processing" containerID="cri-o://cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b" gracePeriod=25 Oct 31 03:47:02.221682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.221532 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" podUID="b26a4824-14ed-448d-90d8-bdf6ec96642a" containerName="payload-pre-processing" containerID="cri-o://66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421" gracePeriod=25 Oct 31 03:47:02.224152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.224114 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:02.235301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.235275 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:47:02.235531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.235519 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:47:02.235680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.235670 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:47:02.235798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.235789 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:47:02.236056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.236041 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:47:02.240175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.237946 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:02.241179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.240797 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:47:02.247091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.241727 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:47:02.247091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.241775 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:47:02.247091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.241810 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:47:02.247091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.241874 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:47:02.247091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.245670 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:47:02.305891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.305103 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:47:02.309368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.309347 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:47:02.314500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.313900 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:47:02.339248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.336405 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 03:47:02.339248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.336496 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 03:47:02.339248 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:02.336709 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa34fb9_c71c_4a6c_b5f5_5df5f4e7a092.slice/crio-32aee9958bd7af60470700d62c4e416f059d771c8c00e6767184f53666281911\": RecentStats: unable to find data in memory cache]" Oct 31 03:47:02.339248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.336752 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:47:02.362326 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.355145 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:47:02.405978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.403595 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:47:02.423238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.423181 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" event={"ID":"7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092","Type":"ContainerDied","Data":"32aee9958bd7af60470700d62c4e416f059d771c8c00e6767184f53666281911"} Oct 31 03:47:02.423444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.423295 5520 scope.go:122] "RemoveContainer" containerID="30dc7b719af2de123c3763ca1af79037d5d5730a951f5898ca4b555d0177a735" Oct 31 03:47:02.423528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.423500 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m" Oct 31 03:47:02.438998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.438956 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" event={"ID":"7e1ca905-81e4-439b-ac0b-facf39d3a7db","Type":"ContainerDied","Data":"6f0edaf51960d87fad6729b5ae45be15e0eef89a3869aea664ffaaaf899db989"} Oct 31 03:47:02.439325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.439307 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm" Oct 31 03:47:02.469972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.469760 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6"] Oct 31 03:47:02.740000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.732921 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n"] Oct 31 03:47:02.740000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.734400 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerName="payload-processing" Oct 31 03:47:02.740000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.734423 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerName="payload-processing" Oct 31 03:47:02.740000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.735615 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerName="payload-pre-processing" Oct 31 03:47:02.740000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.735634 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerName="payload-pre-processing" Oct 31 03:47:02.740000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.735963 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" containerName="payload-pre-processing" Oct 31 03:47:02.740000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.736003 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" containerName="payload-processing" Oct 31 03:47:02.744012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.743848 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:02.748776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.748746 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n"] Oct 31 03:47:02.914834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.914737 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:02.915282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.915259 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wt97\" (UniqueName: \"kubernetes.io/projected/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-kube-api-access-8wt97\") pod \"payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:02.994916 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.994875 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" podUID="b26a4824-14ed-448d-90d8-bdf6ec96642a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:02.995341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:02.995308 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" podUID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:03.000268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.000237 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx"] Oct 31 03:47:03.018224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.014278 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx"] Oct 31 03:47:03.018224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.014434 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.019361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.019334 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m"] Oct 31 03:47:03.020922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.020890 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wt97\" (UniqueName: \"kubernetes.io/projected/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-kube-api-access-8wt97\") pod \"payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:03.021581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.021446 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:03.023962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.023941 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:03.030777 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.030749 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8385c5-77cc7c888c-gj88m"] Oct 31 03:47:03.034891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.034859 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wt97\" (UniqueName: \"kubernetes.io/projected/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-kube-api-access-8wt97\") pod \"payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:03.063515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.063491 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:03.063918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.063837 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:47:03.082482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.082425 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:03.122315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.122240 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm"] Oct 31 03:47:03.129995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.129960 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2gl\" (UniqueName: \"kubernetes.io/projected/f9bdf207-a347-4338-af98-61597014aa0e-kube-api-access-kr2gl\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.130165 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.130024 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.130165 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.130069 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.132496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.132468 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8385c5-7c8d8bd5b5-xm4hm"] Oct 31 03:47:03.233381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.232076 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.233381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.232144 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.233381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.233323 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.234809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.234277 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2gl\" (UniqueName: \"kubernetes.io/projected/f9bdf207-a347-4338-af98-61597014aa0e-kube-api-access-kr2gl\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.243640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.241045 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.261363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.256761 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2gl\" (UniqueName: \"kubernetes.io/projected/f9bdf207-a347-4338-af98-61597014aa0e-kube-api-access-kr2gl\") pod \"maas-api-revoke-sub-abe127c9-hzggx\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.357759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.355941 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:47:03.358903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.358879 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:47:03.375907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.374489 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:47:03.396108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.395853 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:03.430169 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:03.430120 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dbfac7_97ba_468a_b635_ba2a7faf6ef1.slice/crio-09dddb68fcf1edf5ed61f93e7e167f2c6d54f3ef9c9a419519750012571a9e0b WatchSource:0}: Error finding container 09dddb68fcf1edf5ed61f93e7e167f2c6d54f3ef9c9a419519750012571a9e0b: Status 404 returned error can't find the container with id 09dddb68fcf1edf5ed61f93e7e167f2c6d54f3ef9c9a419519750012571a9e0b Oct 31 03:47:03.435291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.435157 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n"] Oct 31 03:47:03.459638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.459596 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" event={"ID":"bd63b361-66b8-4cc5-a8db-619eaf42496b","Type":"ContainerStarted","Data":"f8b29037677fd634a6e3a77138ac18772ce0b21e24fd5600e4bf6d066dec8be5"} Oct 31 03:47:03.459827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.459668 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" event={"ID":"bd63b361-66b8-4cc5-a8db-619eaf42496b","Type":"ContainerStarted","Data":"d64d2df9d2c64bce64b99732cadd51022b4635166270bd9e6239387be5c8a0e8"} Oct 31 03:47:03.461044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.460997 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:47:03.472636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.462725 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" event={"ID":"52dbfac7-97ba-468a-b635-ba2a7faf6ef1","Type":"ContainerStarted","Data":"09dddb68fcf1edf5ed61f93e7e167f2c6d54f3ef9c9a419519750012571a9e0b"} Oct 31 03:47:03.498186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.488976 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" podStartSLOduration=7.488957024 podStartE2EDuration="7.488957024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:03.481194993 +0000 UTC m=+1847.675718116" watchObservedRunningTime="2026-10-31 03:47:03.488957024 +0000 UTC m=+1847.683480145" Oct 31 03:47:03.844883 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:03.844847 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bdf207_a347_4338_af98_61597014aa0e.slice/crio-5e294402cefe2eb0be0ccfdf1d1cf123ec9fb9de660f3b89a28cd4ee5f33a70e WatchSource:0}: Error finding container 5e294402cefe2eb0be0ccfdf1d1cf123ec9fb9de660f3b89a28cd4ee5f33a70e: Status 404 returned error can't find the container with id 5e294402cefe2eb0be0ccfdf1d1cf123ec9fb9de660f3b89a28cd4ee5f33a70e Oct 31 03:47:03.845013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.844924 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx"] Oct 31 03:47:03.922764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.922719 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092" path="/var/lib/kubelet/pods/7aa34fb9-c71c-4a6c-b5f5-5df5f4e7a092/volumes" Oct 31 03:47:03.923607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:03.923581 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1ca905-81e4-439b-ac0b-facf39d3a7db" path="/var/lib/kubelet/pods/7e1ca905-81e4-439b-ac0b-facf39d3a7db/volumes" Oct 31 03:47:04.489250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:04.489100 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" event={"ID":"52dbfac7-97ba-468a-b635-ba2a7faf6ef1","Type":"ContainerStarted","Data":"3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb"} Oct 31 03:47:04.489250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:04.489238 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:04.491263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:04.491232 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" event={"ID":"f9bdf207-a347-4338-af98-61597014aa0e","Type":"ContainerStarted","Data":"490b7ec8d53f7db0013476bf05c5da64e02a293de8b12d770efc4a3d65e4e62c"} Oct 31 03:47:04.491402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:04.491276 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" event={"ID":"f9bdf207-a347-4338-af98-61597014aa0e","Type":"ContainerStarted","Data":"5e294402cefe2eb0be0ccfdf1d1cf123ec9fb9de660f3b89a28cd4ee5f33a70e"} Oct 31 03:47:04.512531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:04.512470 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" podStartSLOduration=8.512452894 podStartE2EDuration="8.512452894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:04.505681998 +0000 UTC m=+1848.700205119" watchObservedRunningTime="2026-10-31 03:47:04.512452894 +0000 UTC m=+1848.706976027" Oct 31 03:47:04.523776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:04.523710 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" podStartSLOduration=2.523689771 podStartE2EDuration="2.523689771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:04.523099468 +0000 UTC m=+1848.717622611" watchObservedRunningTime="2026-10-31 03:47:04.523689771 +0000 UTC m=+1848.718212893" Oct 31 03:47:05.504394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:05.504351 5520 generic.go:356] "Generic (PLEG): container finished" podID="f9bdf207-a347-4338-af98-61597014aa0e" containerID="490b7ec8d53f7db0013476bf05c5da64e02a293de8b12d770efc4a3d65e4e62c" exitCode=0 Oct 31 03:47:05.504961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:05.504562 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" event={"ID":"f9bdf207-a347-4338-af98-61597014aa0e","Type":"ContainerDied","Data":"490b7ec8d53f7db0013476bf05c5da64e02a293de8b12d770efc4a3d65e4e62c"} Oct 31 03:47:06.736434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.736405 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:06.853869 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.853788 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bdf207-a347-4338-af98-61597014aa0e-kube-api-access-kr2gl\" (UniqueName: \"kubernetes.io/projected/f9bdf207-a347-4338-af98-61597014aa0e-kube-api-access-kr2gl\") pod \"f9bdf207-a347-4338-af98-61597014aa0e\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " Oct 31 03:47:06.854023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.853921 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-openshift-ca\") pod \"f9bdf207-a347-4338-af98-61597014aa0e\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " Oct 31 03:47:06.854115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.854099 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-xks-ca\") pod \"f9bdf207-a347-4338-af98-61597014aa0e\" (UID: \"f9bdf207-a347-4338-af98-61597014aa0e\") " Oct 31 03:47:06.854351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.854309 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-openshift-ca" pod "f9bdf207-a347-4338-af98-61597014aa0e" (UID: "f9bdf207-a347-4338-af98-61597014aa0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:06.855835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.855083 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:06.856426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.856400 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bdf207-a347-4338-af98-61597014aa0e-kube-api-access-kr2gl" pod "f9bdf207-a347-4338-af98-61597014aa0e" (UID: "f9bdf207-a347-4338-af98-61597014aa0e"). InnerVolumeSpecName "kube-api-access-kr2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:06.856518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.856437 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-xks-ca" pod "f9bdf207-a347-4338-af98-61597014aa0e" (UID: "f9bdf207-a347-4338-af98-61597014aa0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:06.957046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.957011 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bdf207-a347-4338-af98-61597014aa0e-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:06.957046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:06.957043 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2gl\" (UniqueName: \"kubernetes.io/projected/f9bdf207-a347-4338-af98-61597014aa0e-kube-api-access-kr2gl\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:07.528759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:07.528731 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" Oct 31 03:47:07.528954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:07.528762 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx" event={"ID":"f9bdf207-a347-4338-af98-61597014aa0e","Type":"ContainerDied","Data":"5e294402cefe2eb0be0ccfdf1d1cf123ec9fb9de660f3b89a28cd4ee5f33a70e"} Oct 31 03:47:07.528954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:07.528818 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e294402cefe2eb0be0ccfdf1d1cf123ec9fb9de660f3b89a28cd4ee5f33a70e" Oct 31 03:47:09.442623 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:09.442588 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:47:12.993989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:12.993945 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" podUID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:12.993989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:12.993978 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" podUID="b26a4824-14ed-448d-90d8-bdf6ec96642a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:13.557607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:13.557578 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc"] Oct 31 03:47:14.054497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.054324 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k"] Oct 31 03:47:14.692924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.692888 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv"] Oct 31 03:47:14.693829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.693812 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f9bdf207-a347-4338-af98-61597014aa0e" containerName="revoke-keys" Oct 31 03:47:14.693893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.693831 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bdf207-a347-4338-af98-61597014aa0e" containerName="revoke-keys" Oct 31 03:47:14.694009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.693998 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bdf207-a347-4338-af98-61597014aa0e" containerName="revoke-keys" Oct 31 03:47:14.698689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.698665 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.711967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.711942 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:14.712417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.712392 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:47:14.724428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.722830 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv"] Oct 31 03:47:14.728448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.724919 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.728448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.725280 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.728448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.725349 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntzv\" (UniqueName: \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.827467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.827420 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.827666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.827515 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntzv\" (UniqueName: \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.827666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.827655 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.828687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.828655 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.830414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.830386 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:14.846719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:14.846690 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntzv\" (UniqueName: \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\") pod \"maas-api-revoke-sub-90d978df-thrjv\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:15.015569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:15.015471 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:15.362996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:15.362957 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv"] Oct 31 03:47:15.507519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:15.507486 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:47:15.507686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:15.507559 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:47:15.638773 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:15.638687 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" event={"ID":"67fa4874-2d18-45d4-8f1e-0598742b2c0f","Type":"ContainerStarted","Data":"a95370a257754524b04e00eab9124c037a0291e86b677c65ebf9d35bfe2070ed"} Oct 31 03:47:16.655799 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:16.655761 5520 generic.go:356] "Generic (PLEG): container finished" podID="67fa4874-2d18-45d4-8f1e-0598742b2c0f" containerID="01cefda416b2c97fba45841443f7adfceff65668f52219d552c029b69ffb894a" exitCode=0 Oct 31 03:47:16.656329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:16.655849 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" event={"ID":"67fa4874-2d18-45d4-8f1e-0598742b2c0f","Type":"ContainerDied","Data":"01cefda416b2c97fba45841443f7adfceff65668f52219d552c029b69ffb894a"} Oct 31 03:47:17.663812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:17.663753 5520 scope.go:122] "RemoveContainer" containerID="a6c297918b7b485342a2c3d6bc0a25669f9fa9b1fe86a2332845b167d2421460" Oct 31 03:47:18.226229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.225403 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:18.656313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.656078 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\") pod \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " Oct 31 03:47:18.656502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.656466 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\" (UniqueName: \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\") pod \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " Oct 31 03:47:18.656737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.656717 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\") pod \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " Oct 31 03:47:18.884112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.878762 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca" pod "67fa4874-2d18-45d4-8f1e-0598742b2c0f" (UID: "67fa4874-2d18-45d4-8f1e-0598742b2c0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:18.902930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.891380 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca" pod "67fa4874-2d18-45d4-8f1e-0598742b2c0f" (UID: "67fa4874-2d18-45d4-8f1e-0598742b2c0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:18.938514 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.938142 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv" pod "67fa4874-2d18-45d4-8f1e-0598742b2c0f" (UID: "67fa4874-2d18-45d4-8f1e-0598742b2c0f"). InnerVolumeSpecName "kube-api-access-tntzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:18.949278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.944316 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\" (UniqueName: \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\") pod \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " Oct 31 03:47:18.949278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.944397 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\") pod \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " Oct 31 03:47:18.949278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.948414 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\") pod \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\" (UID: \"67fa4874-2d18-45d4-8f1e-0598742b2c0f\") " Oct 31 03:47:18.950016 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:18.949875 5520 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/67fa4874-2d18-45d4-8f1e-0598742b2c0f/volumes/kubernetes.io~projected/kube-api-access-tntzv Oct 31 03:47:18.950016 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:18.949929 5520 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/67fa4874-2d18-45d4-8f1e-0598742b2c0f/volumes/kubernetes.io~secret/maas-api-xks-ca Oct 31 03:47:18.950016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.949946 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca" pod "67fa4874-2d18-45d4-8f1e-0598742b2c0f" (UID: "67fa4874-2d18-45d4-8f1e-0598742b2c0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:18.950016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.949961 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv" pod "67fa4874-2d18-45d4-8f1e-0598742b2c0f" (UID: "67fa4874-2d18-45d4-8f1e-0598742b2c0f"). InnerVolumeSpecName "kube-api-access-tntzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:18.950016 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:18.949878 5520 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/67fa4874-2d18-45d4-8f1e-0598742b2c0f/volumes/kubernetes.io~configmap/maas-api-openshift-ca Oct 31 03:47:18.950016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.949988 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca" pod "67fa4874-2d18-45d4-8f1e-0598742b2c0f" (UID: "67fa4874-2d18-45d4-8f1e-0598742b2c0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:18.971409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.959639 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntzv\" (UniqueName: \"kubernetes.io/projected/67fa4874-2d18-45d4-8f1e-0598742b2c0f-kube-api-access-tntzv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:18.971409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.959672 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:18.971409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:18.959687 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67fa4874-2d18-45d4-8f1e-0598742b2c0f-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:19.001713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:19.001670 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" Oct 31 03:47:19.002094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:19.002065 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv" event={"ID":"67fa4874-2d18-45d4-8f1e-0598742b2c0f","Type":"ContainerDied","Data":"a95370a257754524b04e00eab9124c037a0291e86b677c65ebf9d35bfe2070ed"} Oct 31 03:47:19.002273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:19.002255 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a95370a257754524b04e00eab9124c037a0291e86b677c65ebf9d35bfe2070ed" Oct 31 03:47:22.625104 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.625078 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:47:22.686702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.686640 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-plugins-config-volume\") pod \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " Oct 31 03:47:22.687255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.687172 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-kube-api-access-qxc5p\" (UniqueName: \"kubernetes.io/projected/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-kube-api-access-qxc5p\") pod \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\" (UID: \"cf6465fc-e357-4c38-8cd0-c12fb7f45dce\") " Oct 31 03:47:22.688587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.688557 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-plugins-config-volume" pod "cf6465fc-e357-4c38-8cd0-c12fb7f45dce" (UID: "cf6465fc-e357-4c38-8cd0-c12fb7f45dce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:22.690185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.690156 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:47:22.691163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.691134 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-kube-api-access-qxc5p" pod "cf6465fc-e357-4c38-8cd0-c12fb7f45dce" (UID: "cf6465fc-e357-4c38-8cd0-c12fb7f45dce"). InnerVolumeSpecName "kube-api-access-qxc5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:22.720568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.720538 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x"] Oct 31 03:47:22.721471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721448 5520 cpu_manager.go:414] "containerMap: removing container" podUID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" containerName="payload-processing" Oct 31 03:47:22.721471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721467 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" containerName="payload-processing" Oct 31 03:47:22.721649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721500 5520 cpu_manager.go:414] "containerMap: removing container" podUID="67fa4874-2d18-45d4-8f1e-0598742b2c0f" containerName="revoke-keys" Oct 31 03:47:22.721649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721505 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fa4874-2d18-45d4-8f1e-0598742b2c0f" containerName="revoke-keys" Oct 31 03:47:22.721649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721516 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b26a4824-14ed-448d-90d8-bdf6ec96642a" containerName="payload-pre-processing" Oct 31 03:47:22.721649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721521 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26a4824-14ed-448d-90d8-bdf6ec96642a" containerName="payload-pre-processing" Oct 31 03:47:22.721848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721665 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fa4874-2d18-45d4-8f1e-0598742b2c0f" containerName="revoke-keys" Oct 31 03:47:22.721848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721677 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26a4824-14ed-448d-90d8-bdf6ec96642a" containerName="payload-pre-processing" Oct 31 03:47:22.721848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.721692 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" containerName="payload-processing" Oct 31 03:47:22.725684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.725660 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.736229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.732478 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b8fd22-openshift-default-dockercfg-h65mt\"" Oct 31 03:47:22.761619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.761571 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x"] Oct 31 03:47:22.789891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.789857 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26a4824-14ed-448d-90d8-bdf6ec96642a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26a4824-14ed-448d-90d8-bdf6ec96642a-plugins-config-volume\") pod \"b26a4824-14ed-448d-90d8-bdf6ec96642a\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " Oct 31 03:47:22.790081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.789915 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26a4824-14ed-448d-90d8-bdf6ec96642a-kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/b26a4824-14ed-448d-90d8-bdf6ec96642a-kube-api-access-7src8\") pod \"b26a4824-14ed-448d-90d8-bdf6ec96642a\" (UID: \"b26a4824-14ed-448d-90d8-bdf6ec96642a\") " Oct 31 03:47:22.790746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.790608 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26a4824-14ed-448d-90d8-bdf6ec96642a-plugins-config-volume" pod "b26a4824-14ed-448d-90d8-bdf6ec96642a" (UID: "b26a4824-14ed-448d-90d8-bdf6ec96642a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:22.791730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.791696 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-credential-socket\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.791864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.791758 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-certs\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.791864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.791820 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istiod-ca-cert\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.791986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.791941 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-token\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.792042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.791983 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-data\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.792096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.792064 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-envoy\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.792148 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.792102 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-socket\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.792214 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.792176 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2hlh\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-kube-api-access-q2hlh\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.792865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.792401 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-podinfo\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.793045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.793024 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxc5p\" (UniqueName: \"kubernetes.io/projected/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-kube-api-access-qxc5p\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:22.793119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.793056 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf6465fc-e357-4c38-8cd0-c12fb7f45dce-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:22.793119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.793072 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b26a4824-14ed-448d-90d8-bdf6ec96642a-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:22.801088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.801050 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26a4824-14ed-448d-90d8-bdf6ec96642a-kube-api-access-7src8" pod "b26a4824-14ed-448d-90d8-bdf6ec96642a" (UID: "b26a4824-14ed-448d-90d8-bdf6ec96642a"). InnerVolumeSpecName "kube-api-access-7src8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:22.920292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.910113 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-credential-socket\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.934477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.934443 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-certs\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.934797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.934773 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istiod-ca-cert\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.935053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.935036 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-token\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.945817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.937841 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-certs\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.945817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.915560 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-credential-socket\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.945817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.938579 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istiod-ca-cert\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.946243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.946215 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-data\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.966417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.966377 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-envoy\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.966692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.966664 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-socket\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.967374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.967336 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2hlh\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-kube-api-access-q2hlh\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.967595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.967579 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-podinfo\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.968092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.946700 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-data\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.968173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.967373 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-socket\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.968301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.968285 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7src8\" (UniqueName: \"kubernetes.io/projected/b26a4824-14ed-448d-90d8-bdf6ec96642a-kube-api-access-7src8\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:22.980994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.977797 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-podinfo\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.983035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.982976 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-envoy\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.983944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.983912 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2hlh\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-kube-api-access-q2hlh\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:22.984409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:22.984343 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-token\") pod \"e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:23.046815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.046780 5520 generic.go:356] "Generic (PLEG): container finished" podID="b26a4824-14ed-448d-90d8-bdf6ec96642a" containerID="66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421" exitCode=0 Oct 31 03:47:23.047019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.046847 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" Oct 31 03:47:23.047019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.046869 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" event={"ID":"b26a4824-14ed-448d-90d8-bdf6ec96642a","Type":"ContainerDied","Data":"66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421"} Oct 31 03:47:23.047019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.046931 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j" event={"ID":"b26a4824-14ed-448d-90d8-bdf6ec96642a","Type":"ContainerDied","Data":"22d9a27e18cad1983dafd67ab2c9ea6135c6875709c4ec6bb7ae489017dd930a"} Oct 31 03:47:23.047019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.046945 5520 scope.go:122] "RemoveContainer" containerID="66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421" Oct 31 03:47:23.048074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.048050 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:23.048894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.048861 5520 generic.go:356] "Generic (PLEG): container finished" podID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" containerID="cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b" exitCode=0 Oct 31 03:47:23.048983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.048915 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" Oct 31 03:47:23.049043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.048918 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" event={"ID":"cf6465fc-e357-4c38-8cd0-c12fb7f45dce","Type":"ContainerDied","Data":"cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b"} Oct 31 03:47:23.049087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.049053 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh" event={"ID":"cf6465fc-e357-4c38-8cd0-c12fb7f45dce","Type":"ContainerDied","Data":"a2e44a4199a2fd4cdff64132e30b1342f63692d809adacff6661a6ee773a8988"} Oct 31 03:47:23.079479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.079437 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j"] Oct 31 03:47:23.091568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.091537 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6f28b4-68f7858b79-x2t2j"] Oct 31 03:47:23.091722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.091699 5520 scope.go:122] "RemoveContainer" containerID="66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421" Oct 31 03:47:23.092110 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:23.092084 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421\": container with ID starting with 66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421 not found: ID does not exist" containerID="66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421" Oct 31 03:47:23.092189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.092123 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421"} err="failed to get container status \"66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421\": rpc error: code = NotFound desc = could not find container \"66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421\": container with ID starting with 66e37999ffb8a79f395349a5ee572fe607790df3d481f945990a2e1fd5602421 not found: ID does not exist" Oct 31 03:47:23.092189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.092148 5520 scope.go:122] "RemoveContainer" containerID="cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b" Oct 31 03:47:23.108680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.108650 5520 scope.go:122] "RemoveContainer" containerID="cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b" Oct 31 03:47:23.108895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.108873 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh"] Oct 31 03:47:23.109127 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:23.109090 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b\": container with ID starting with cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b not found: ID does not exist" containerID="cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b" Oct 31 03:47:23.109251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.109134 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b"} err="failed to get container status \"cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b\": rpc error: code = NotFound desc = could not find container \"cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b\": container with ID starting with cb7301839386950f5fabcc8a8b0ba72dd89c0fcebac4f793e8bb9261619a541b not found: ID does not exist" Oct 31 03:47:23.121320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.121292 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f28b4-64b4d869dc-vk4lh"] Oct 31 03:47:23.372269 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:23.372177 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb74b35_d0cc_4865_9617_263b41b4ec7a.slice/crio-48fe2fa30016c545b68079ec6748dacbee70f659b2757a5a2bc8a8773065b852 WatchSource:0}: Error finding container 48fe2fa30016c545b68079ec6748dacbee70f659b2757a5a2bc8a8773065b852: Status 404 returned error can't find the container with id 48fe2fa30016c545b68079ec6748dacbee70f659b2757a5a2bc8a8773065b852 Oct 31 03:47:23.373406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.373368 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x"] Oct 31 03:47:23.379037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.378751 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:47:23.379037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.378846 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:47:23.379037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.378889 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:47:23.926345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.926298 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26a4824-14ed-448d-90d8-bdf6ec96642a" path="/var/lib/kubelet/pods/b26a4824-14ed-448d-90d8-bdf6ec96642a/volumes" Oct 31 03:47:23.927106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:23.927083 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6465fc-e357-4c38-8cd0-c12fb7f45dce" path="/var/lib/kubelet/pods/cf6465fc-e357-4c38-8cd0-c12fb7f45dce/volumes" Oct 31 03:47:24.063287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:24.063255 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" event={"ID":"fcb74b35-d0cc-4865-9617-263b41b4ec7a","Type":"ContainerStarted","Data":"75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8"} Oct 31 03:47:24.063456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:24.063313 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" event={"ID":"fcb74b35-d0cc-4865-9617-263b41b4ec7a","Type":"ContainerStarted","Data":"48fe2fa30016c545b68079ec6748dacbee70f659b2757a5a2bc8a8773065b852"} Oct 31 03:47:25.048603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:25.048555 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:25.055599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:25.055570 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:25.075337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:25.075293 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:25.076395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:25.076372 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:47:25.083385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:25.083335 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podStartSLOduration=6.083322206 podStartE2EDuration="6.083322206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:24.099414282 +0000 UTC m=+1868.293937417" watchObservedRunningTime="2026-10-31 03:47:25.083322206 +0000 UTC m=+1869.277845326" Oct 31 03:47:27.897228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:27.896747 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb"] Oct 31 03:47:28.648228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:28.644529 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w"] Oct 31 03:47:28.979227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:28.971964 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b"] Oct 31 03:47:29.005227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:28.999414 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.016234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.011824 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:47:29.016234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.011935 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:47:29.118233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.112646 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.118233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.112878 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.118233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.112970 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjbw\" (UniqueName: \"kubernetes.io/projected/aec13ba1-1656-4f86-99be-56aa932a2e2e-kube-api-access-2hjbw\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.144231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.143023 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b"] Oct 31 03:47:29.224236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.218970 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.224236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.219151 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.224236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.219238 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjbw\" (UniqueName: \"kubernetes.io/projected/aec13ba1-1656-4f86-99be-56aa932a2e2e-kube-api-access-2hjbw\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.224236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.220692 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.224236 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.224046 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.243273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.236414 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" containerID="cri-o://526c0e425966e82aea7da7b69143273257290df6a68cac9a2bec8f8077d8ec78" gracePeriod=45 Oct 31 03:47:29.255915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.255878 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" containerID="cri-o://1519da2c89be388e2db8aab4993f471cac6af9c6133d7a6ab33b5a2c52d012b5" gracePeriod=45 Oct 31 03:47:29.258028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.256141 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": EOF" Oct 31 03:47:29.306377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.306130 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjbw\" (UniqueName: \"kubernetes.io/projected/aec13ba1-1656-4f86-99be-56aa932a2e2e-kube-api-access-2hjbw\") pod \"maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.307123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.306673 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": dial tcp 10.134.0.169:8000: connect: connection refused" Oct 31 03:47:29.307123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.306789 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:47:29.336275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.336230 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": dial tcp 10.134.0.167:8000: connect: connection refused" Oct 31 03:47:29.387692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.387664 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:29.487967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.487381 5520 generic.go:356] "Generic (PLEG): container finished" podID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerID="1519da2c89be388e2db8aab4993f471cac6af9c6133d7a6ab33b5a2c52d012b5" exitCode=0 Oct 31 03:47:29.487967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.487610 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" event={"ID":"d3a35073-afca-4267-aa1b-4c8fe008f6b5","Type":"ContainerDied","Data":"1519da2c89be388e2db8aab4993f471cac6af9c6133d7a6ab33b5a2c52d012b5"} Oct 31 03:47:29.489685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.489616 5520 generic.go:356] "Generic (PLEG): container finished" podID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerID="526c0e425966e82aea7da7b69143273257290df6a68cac9a2bec8f8077d8ec78" exitCode=0 Oct 31 03:47:29.489787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.489693 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" event={"ID":"ed3bfa65-ac7b-4b46-81d3-8319d1f54937","Type":"ContainerDied","Data":"526c0e425966e82aea7da7b69143273257290df6a68cac9a2bec8f8077d8ec78"} Oct 31 03:47:29.760821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.760788 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:47:29.775853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.775824 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:47:29.852372 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852296 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-dshm\") pod \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " Oct 31 03:47:29.852532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852402 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-model-cache\") pod \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " Oct 31 03:47:29.852573 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852541 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-kube-api-access-2qk65\" (UniqueName: \"kubernetes.io/projected/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-kube-api-access-2qk65\") pod \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " Oct 31 03:47:29.852611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852579 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-dshm\") pod \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " Oct 31 03:47:29.852688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852672 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tls-certs\") pod \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " Oct 31 03:47:29.852749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852688 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-model-cache" pod "d3a35073-afca-4267-aa1b-4c8fe008f6b5" (UID: "d3a35073-afca-4267-aa1b-4c8fe008f6b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.852749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852742 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-home\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-home\") pod \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " Oct 31 03:47:29.852865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852773 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tmp-dir\") pod \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " Oct 31 03:47:29.852865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.852814 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tls-certs\") pod \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " Oct 31 03:47:29.853143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853117 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-home" pod "ed3bfa65-ac7b-4b46-81d3-8319d1f54937" (UID: "ed3bfa65-ac7b-4b46-81d3-8319d1f54937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.853481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853180 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tmp-dir" pod "ed3bfa65-ac7b-4b46-81d3-8319d1f54937" (UID: "ed3bfa65-ac7b-4b46-81d3-8319d1f54937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.853481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853311 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-home\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-home\") pod \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " Oct 31 03:47:29.853481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853352 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a35073-afca-4267-aa1b-4c8fe008f6b5-kube-api-access-q2ddv\" (UniqueName: \"kubernetes.io/projected/d3a35073-afca-4267-aa1b-4c8fe008f6b5-kube-api-access-q2ddv\") pod \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " Oct 31 03:47:29.853481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853374 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tmp-dir\") pod \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\" (UID: \"d3a35073-afca-4267-aa1b-4c8fe008f6b5\") " Oct 31 03:47:29.853481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853484 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-model-cache\") pod \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\" (UID: \"ed3bfa65-ac7b-4b46-81d3-8319d1f54937\") " Oct 31 03:47:29.853784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853608 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-home" pod "d3a35073-afca-4267-aa1b-4c8fe008f6b5" (UID: "d3a35073-afca-4267-aa1b-4c8fe008f6b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.853844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.853807 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tmp-dir" pod "d3a35073-afca-4267-aa1b-4c8fe008f6b5" (UID: "d3a35073-afca-4267-aa1b-4c8fe008f6b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.854098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.854073 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-model-cache" pod "ed3bfa65-ac7b-4b46-81d3-8319d1f54937" (UID: "ed3bfa65-ac7b-4b46-81d3-8319d1f54937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.854816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.854772 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.854816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.854801 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.854816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.854814 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.854963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.854827 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.854963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.854841 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.854963 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.854854 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.855433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.855412 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tls-certs" pod "d3a35073-afca-4267-aa1b-4c8fe008f6b5" (UID: "d3a35073-afca-4267-aa1b-4c8fe008f6b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:29.855585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.855558 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-dshm" pod "d3a35073-afca-4267-aa1b-4c8fe008f6b5" (UID: "d3a35073-afca-4267-aa1b-4c8fe008f6b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.855695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.855586 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-dshm" pod "ed3bfa65-ac7b-4b46-81d3-8319d1f54937" (UID: "ed3bfa65-ac7b-4b46-81d3-8319d1f54937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:29.855826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.855809 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-kube-api-access-2qk65" pod "ed3bfa65-ac7b-4b46-81d3-8319d1f54937" (UID: "ed3bfa65-ac7b-4b46-81d3-8319d1f54937"). InnerVolumeSpecName "kube-api-access-2qk65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:29.855890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.855853 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a35073-afca-4267-aa1b-4c8fe008f6b5-kube-api-access-q2ddv" pod "d3a35073-afca-4267-aa1b-4c8fe008f6b5" (UID: "d3a35073-afca-4267-aa1b-4c8fe008f6b5"). InnerVolumeSpecName "kube-api-access-q2ddv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:29.856674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.856653 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tls-certs" pod "ed3bfa65-ac7b-4b46-81d3-8319d1f54937" (UID: "ed3bfa65-ac7b-4b46-81d3-8319d1f54937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:29.920434 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:29.920157 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec13ba1_1656_4f86_99be_56aa932a2e2e.slice/crio-325dde9ec13c2a4bc71edf1ac31c7e69cc9bd5594341f147176915b88c6e0c08 WatchSource:0}: Error finding container 325dde9ec13c2a4bc71edf1ac31c7e69cc9bd5594341f147176915b88c6e0c08: Status 404 returned error can't find the container with id 325dde9ec13c2a4bc71edf1ac31c7e69cc9bd5594341f147176915b88c6e0c08 Oct 31 03:47:29.925862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.925837 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b"] Oct 31 03:47:29.957774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.957746 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a35073-afca-4267-aa1b-4c8fe008f6b5-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.957881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.957776 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qk65\" (UniqueName: \"kubernetes.io/projected/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-kube-api-access-2qk65\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.957881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.957790 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.957881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.957798 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a35073-afca-4267-aa1b-4c8fe008f6b5-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.957881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.957806 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed3bfa65-ac7b-4b46-81d3-8319d1f54937-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:29.957881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:29.957815 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ddv\" (UniqueName: \"kubernetes.io/projected/d3a35073-afca-4267-aa1b-4c8fe008f6b5-kube-api-access-q2ddv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:30.502343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.502238 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" event={"ID":"d3a35073-afca-4267-aa1b-4c8fe008f6b5","Type":"ContainerDied","Data":"64a08152c101e2fc0cca833eb9f170c7ba4cd4efd8e7a72cd4934655d1d840f5"} Oct 31 03:47:30.502343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.502291 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k" Oct 31 03:47:30.502866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.502300 5520 scope.go:122] "RemoveContainer" containerID="1519da2c89be388e2db8aab4993f471cac6af9c6133d7a6ab33b5a2c52d012b5" Oct 31 03:47:30.508242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.508194 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" event={"ID":"aec13ba1-1656-4f86-99be-56aa932a2e2e","Type":"ContainerStarted","Data":"3f8844757b61bcb061933d4bed2b83b7f90e1ceb08aa0da2887d76dd57e348f8"} Oct 31 03:47:30.508376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.508309 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" event={"ID":"aec13ba1-1656-4f86-99be-56aa932a2e2e","Type":"ContainerStarted","Data":"325dde9ec13c2a4bc71edf1ac31c7e69cc9bd5594341f147176915b88c6e0c08"} Oct 31 03:47:30.510094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.510064 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" event={"ID":"ed3bfa65-ac7b-4b46-81d3-8319d1f54937","Type":"ContainerDied","Data":"f5a151ccc82467929b5868ea38dd9f5239726106aeead6642db2930f9e95ff8f"} Oct 31 03:47:30.510254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.510163 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc" Oct 31 03:47:30.516983 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.516957 5520 scope.go:122] "RemoveContainer" containerID="526c0e425966e82aea7da7b69143273257290df6a68cac9a2bec8f8077d8ec78" Oct 31 03:47:30.526230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.526172 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k"] Oct 31 03:47:30.532318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.532290 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b2dc82/test-model-b2dc82-kserve-76d94b47c7-b2w5k"] Oct 31 03:47:30.537660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.537636 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc"] Oct 31 03:47:30.542406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.542384 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c5dcb7/test-model-c5dcb7-kserve-5d6c9bd5ff-fj7rc"] Oct 31 03:47:30.550932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:30.550886 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" podStartSLOduration=2.550873652 podStartE2EDuration="2.550873652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:30.547880219 +0000 UTC m=+1874.742403383" watchObservedRunningTime="2026-10-31 03:47:30.550873652 +0000 UTC m=+1874.745396773" Oct 31 03:47:31.617502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:31.617280 5520 generic.go:356] "Generic (PLEG): container finished" podID="aec13ba1-1656-4f86-99be-56aa932a2e2e" containerID="3f8844757b61bcb061933d4bed2b83b7f90e1ceb08aa0da2887d76dd57e348f8" exitCode=0 Oct 31 03:47:31.617502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:31.617481 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" event={"ID":"aec13ba1-1656-4f86-99be-56aa932a2e2e","Type":"ContainerDied","Data":"3f8844757b61bcb061933d4bed2b83b7f90e1ceb08aa0da2887d76dd57e348f8"} Oct 31 03:47:31.922587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:31.922513 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" path="/var/lib/kubelet/pods/d3a35073-afca-4267-aa1b-4c8fe008f6b5/volumes" Oct 31 03:47:31.923037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:31.923022 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" path="/var/lib/kubelet/pods/ed3bfa65-ac7b-4b46-81d3-8319d1f54937/volumes" Oct 31 03:47:32.852914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:32.852885 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:32.956226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:32.949011 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-openshift-ca\") pod \"aec13ba1-1656-4f86-99be-56aa932a2e2e\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " Oct 31 03:47:32.956226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:32.949494 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec13ba1-1656-4f86-99be-56aa932a2e2e-kube-api-access-2hjbw\" (UniqueName: \"kubernetes.io/projected/aec13ba1-1656-4f86-99be-56aa932a2e2e-kube-api-access-2hjbw\") pod \"aec13ba1-1656-4f86-99be-56aa932a2e2e\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " Oct 31 03:47:32.956226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:32.949609 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-xks-ca\") pod \"aec13ba1-1656-4f86-99be-56aa932a2e2e\" (UID: \"aec13ba1-1656-4f86-99be-56aa932a2e2e\") " Oct 31 03:47:32.993223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:32.986676 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-openshift-ca" pod "aec13ba1-1656-4f86-99be-56aa932a2e2e" (UID: "aec13ba1-1656-4f86-99be-56aa932a2e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:32.993223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:32.989005 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-xks-ca" pod "aec13ba1-1656-4f86-99be-56aa932a2e2e" (UID: "aec13ba1-1656-4f86-99be-56aa932a2e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:33.004875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:33.004803 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec13ba1-1656-4f86-99be-56aa932a2e2e-kube-api-access-2hjbw" pod "aec13ba1-1656-4f86-99be-56aa932a2e2e" (UID: "aec13ba1-1656-4f86-99be-56aa932a2e2e"). InnerVolumeSpecName "kube-api-access-2hjbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:33.055049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:33.055015 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:33.055049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:33.055045 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hjbw\" (UniqueName: \"kubernetes.io/projected/aec13ba1-1656-4f86-99be-56aa932a2e2e-kube-api-access-2hjbw\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:33.055049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:33.055056 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec13ba1-1656-4f86-99be-56aa932a2e2e-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:33.640737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:33.640702 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" Oct 31 03:47:33.640913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:33.640726 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b" event={"ID":"aec13ba1-1656-4f86-99be-56aa932a2e2e","Type":"ContainerDied","Data":"325dde9ec13c2a4bc71edf1ac31c7e69cc9bd5594341f147176915b88c6e0c08"} Oct 31 03:47:33.640913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:33.640811 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="325dde9ec13c2a4bc71edf1ac31c7e69cc9bd5594341f147176915b88c6e0c08" Oct 31 03:47:36.151688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:36.151651 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr"] Oct 31 03:47:36.152168 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:36.151861 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" containerID="cri-o://b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd" gracePeriod=30 Oct 31 03:47:38.927687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:38.927643 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q"] Oct 31 03:47:38.975625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:38.975589 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk"] Oct 31 03:47:39.393566 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:39.393528 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:47:41.251993 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:41.251957 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e030301_62ab_4734_9bcf_cd9a0854817c.slice/crio-b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e030301_62ab_4734_9bcf_cd9a0854817c.slice/crio-conmon-b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:47:41.550264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.550227 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:47:41.642708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.642678 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-credential-socket\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.642708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.642722 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-kube-api-access-9rjkv\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-kube-api-access-9rjkv\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.642951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.642747 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e030301-62ab-4734-9bcf-cd9a0854817c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e030301-62ab-4734-9bcf-cd9a0854817c-istiod-ca-cert\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.642951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.642801 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-certs\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.643101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.642968 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-credential-socket" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:41.643101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.642989 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-envoy\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.643101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643029 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-token\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-token\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.643354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643240 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-podinfo\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.643354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643247 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e030301-62ab-4734-9bcf-cd9a0854817c-istiod-ca-cert" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:41.643354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643333 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-socket\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.643529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643502 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-certs" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:41.643674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643624 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-socket" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:41.643828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643812 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-data\") pod \"2e030301-62ab-4734-9bcf-cd9a0854817c\" (UID: \"2e030301-62ab-4734-9bcf-cd9a0854817c\") " Oct 31 03:47:41.643955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.643931 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-data" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:41.644817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.644787 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.644817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.644814 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.644973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.644830 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.644973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.644842 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e030301-62ab-4734-9bcf-cd9a0854817c-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.644973 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.644854 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.645721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.645694 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-kube-api-access-9rjkv" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "kube-api-access-9rjkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:41.645829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.645809 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-podinfo" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:47:41.645979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.645960 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-token" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:41.646588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.646568 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-envoy" pod "2e030301-62ab-4734-9bcf-cd9a0854817c" (UID: "2e030301-62ab-4734-9bcf-cd9a0854817c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:41.774121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.752719 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.775400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.775373 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rjkv\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-kube-api-access-9rjkv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.775571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.775557 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.780468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.776421 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e030301-62ab-4734-9bcf-cd9a0854817c-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:41.936669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.936626 5520 generic.go:356] "Generic (PLEG): container finished" podID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerID="b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd" exitCode=0 Oct 31 03:47:41.936820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.936674 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" Oct 31 03:47:41.936820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.936726 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" event={"ID":"2e030301-62ab-4734-9bcf-cd9a0854817c","Type":"ContainerDied","Data":"b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd"} Oct 31 03:47:41.936820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.936797 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr" event={"ID":"2e030301-62ab-4734-9bcf-cd9a0854817c","Type":"ContainerDied","Data":"c12ee05bc9307ae9bea481af64697d654167b7b653748d6cd72b72ca957d7517"} Oct 31 03:47:41.936993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.936828 5520 scope.go:122] "RemoveContainer" containerID="b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd" Oct 31 03:47:41.950410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.950393 5520 scope.go:122] "RemoveContainer" containerID="b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd" Oct 31 03:47:41.950686 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:41.950668 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd\": container with ID starting with b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd not found: ID does not exist" containerID="b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd" Oct 31 03:47:41.950762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.950691 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd"} err="failed to get container status \"b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd\": rpc error: code = NotFound desc = could not find container \"b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd\": container with ID starting with b83687f6a35ca7b840687ae302c92acc6f2fabcb26af2796a677b2b354e148fd not found: ID does not exist" Oct 31 03:47:41.959194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.959168 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr"] Oct 31 03:47:41.966183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:41.966159 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-gh6nr"] Oct 31 03:47:42.938540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:42.938499 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" containerID="cri-o://7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89" gracePeriod=45 Oct 31 03:47:43.262176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.262144 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:47:43.314280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.314244 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-home\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-home\") pod \"0694d1ee-0802-47de-a550-53f1da6e183b\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " Oct 31 03:47:43.314994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.314531 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0694d1ee-0802-47de-a550-53f1da6e183b-kube-api-access-4jgrv\" (UniqueName: \"kubernetes.io/projected/0694d1ee-0802-47de-a550-53f1da6e183b-kube-api-access-4jgrv\") pod \"0694d1ee-0802-47de-a550-53f1da6e183b\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " Oct 31 03:47:43.315314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.315296 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0694d1ee-0802-47de-a550-53f1da6e183b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0694d1ee-0802-47de-a550-53f1da6e183b-tls-certs\") pod \"0694d1ee-0802-47de-a550-53f1da6e183b\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " Oct 31 03:47:43.315506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.315344 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-home" pod "0694d1ee-0802-47de-a550-53f1da6e183b" (UID: "0694d1ee-0802-47de-a550-53f1da6e183b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:43.315851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.315837 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-dshm\") pod \"0694d1ee-0802-47de-a550-53f1da6e183b\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " Oct 31 03:47:43.316375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.316356 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-model-cache\") pod \"0694d1ee-0802-47de-a550-53f1da6e183b\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " Oct 31 03:47:43.316964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.316806 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-tmp-dir\") pod \"0694d1ee-0802-47de-a550-53f1da6e183b\" (UID: \"0694d1ee-0802-47de-a550-53f1da6e183b\") " Oct 31 03:47:43.317234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.316744 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-model-cache" pod "0694d1ee-0802-47de-a550-53f1da6e183b" (UID: "0694d1ee-0802-47de-a550-53f1da6e183b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:43.317234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.317101 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-tmp-dir" pod "0694d1ee-0802-47de-a550-53f1da6e183b" (UID: "0694d1ee-0802-47de-a550-53f1da6e183b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:43.318074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.318048 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0694d1ee-0802-47de-a550-53f1da6e183b-tls-certs" pod "0694d1ee-0802-47de-a550-53f1da6e183b" (UID: "0694d1ee-0802-47de-a550-53f1da6e183b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:43.318601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.318579 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:43.318722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.318597 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0694d1ee-0802-47de-a550-53f1da6e183b-kube-api-access-4jgrv" pod "0694d1ee-0802-47de-a550-53f1da6e183b" (UID: "0694d1ee-0802-47de-a550-53f1da6e183b"). InnerVolumeSpecName "kube-api-access-4jgrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:43.318722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.318605 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0694d1ee-0802-47de-a550-53f1da6e183b-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:43.318722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.318648 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:43.318722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.318665 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:43.320698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.320675 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-dshm" pod "0694d1ee-0802-47de-a550-53f1da6e183b" (UID: "0694d1ee-0802-47de-a550-53f1da6e183b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:43.447230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.446956 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0694d1ee-0802-47de-a550-53f1da6e183b-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:43.447230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.447021 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jgrv\" (UniqueName: \"kubernetes.io/projected/0694d1ee-0802-47de-a550-53f1da6e183b-kube-api-access-4jgrv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:43.928163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.928098 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" podUID="cc412090-34af-455d-b359-890af2e14f43" containerName="payload-processing" containerID="cri-o://9a4c6560473e28db8b728935ab306f48cf14cd8f734a72d823b9f8a6afed684e" gracePeriod=25 Oct 31 03:47:43.931864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.931837 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" path="/var/lib/kubelet/pods/2e030301-62ab-4734-9bcf-cd9a0854817c/volumes" Oct 31 03:47:43.958934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.958909 5520 generic.go:356] "Generic (PLEG): container finished" podID="0694d1ee-0802-47de-a550-53f1da6e183b" containerID="7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89" exitCode=0 Oct 31 03:47:43.959400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.958964 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" Oct 31 03:47:43.959400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.959001 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" event={"ID":"0694d1ee-0802-47de-a550-53f1da6e183b","Type":"ContainerDied","Data":"7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89"} Oct 31 03:47:43.959400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.959050 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb" event={"ID":"0694d1ee-0802-47de-a550-53f1da6e183b","Type":"ContainerDied","Data":"d01606cdf0f2796817c10a460ae63b7dac95f93d947db4ea1c9a77fa7afff9b8"} Oct 31 03:47:43.959400 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.959080 5520 scope.go:122] "RemoveContainer" containerID="7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89" Oct 31 03:47:43.973415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.973383 5520 scope.go:122] "RemoveContainer" containerID="7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89" Oct 31 03:47:43.973751 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:43.973724 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89\": container with ID starting with 7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89 not found: ID does not exist" containerID="7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89" Oct 31 03:47:43.973850 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.973757 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89"} err="failed to get container status \"7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89\": rpc error: code = NotFound desc = could not find container \"7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89\": container with ID starting with 7a664e2f4510277ed478844b347978b52a060ad09d16aee86946183ac044dc89 not found: ID does not exist" Oct 31 03:47:43.976476 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.976452 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" podUID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" containerName="payload-pre-processing" containerID="cri-o://f6b5bfa763e3c426cc7125e888b4a90f0d79eac5eb3ed5a37ef98c305be9912e" gracePeriod=25 Oct 31 03:47:43.987215 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.987174 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb"] Oct 31 03:47:43.994453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:43.994428 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-73e222/test-model-73e222-kserve-59bd9c5445-fldxb"] Oct 31 03:47:44.321297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.321232 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" containerID="cri-o://383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217" gracePeriod=45 Oct 31 03:47:44.641686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.641655 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:47:44.773838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.773799 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-kube-api-access-mdm6b\" (UniqueName: \"kubernetes.io/projected/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-kube-api-access-mdm6b\") pod \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " Oct 31 03:47:44.775762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.775736 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tls-certs\") pod \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " Oct 31 03:47:44.776502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.776481 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tmp-dir\") pod \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " Oct 31 03:47:44.777076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.776892 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tmp-dir" pod "c04bd3bb-7308-4d36-95d1-d35a1d8b431b" (UID: "c04bd3bb-7308-4d36-95d1-d35a1d8b431b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:44.777382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.777362 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-home\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-home\") pod \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " Oct 31 03:47:44.777796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.777777 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-dshm\") pod \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " Oct 31 03:47:44.778432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.778412 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-model-cache\") pod \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\" (UID: \"c04bd3bb-7308-4d36-95d1-d35a1d8b431b\") " Oct 31 03:47:44.779911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.779888 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:44.780056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.777675 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-home" pod "c04bd3bb-7308-4d36-95d1-d35a1d8b431b" (UID: "c04bd3bb-7308-4d36-95d1-d35a1d8b431b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:44.780153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.778826 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-model-cache" pod "c04bd3bb-7308-4d36-95d1-d35a1d8b431b" (UID: "c04bd3bb-7308-4d36-95d1-d35a1d8b431b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:44.784771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.784719 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tls-certs" pod "c04bd3bb-7308-4d36-95d1-d35a1d8b431b" (UID: "c04bd3bb-7308-4d36-95d1-d35a1d8b431b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:44.806931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.806612 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-kube-api-access-mdm6b" pod "c04bd3bb-7308-4d36-95d1-d35a1d8b431b" (UID: "c04bd3bb-7308-4d36-95d1-d35a1d8b431b"). InnerVolumeSpecName "kube-api-access-mdm6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:44.808275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.807327 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-dshm" pod "c04bd3bb-7308-4d36-95d1-d35a1d8b431b" (UID: "c04bd3bb-7308-4d36-95d1-d35a1d8b431b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:44.881732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.881658 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:44.881732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.881688 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdm6b\" (UniqueName: \"kubernetes.io/projected/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-kube-api-access-mdm6b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:44.881732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.881705 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:44.881732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.881714 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:44.881732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.881724 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c04bd3bb-7308-4d36-95d1-d35a1d8b431b-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:44.970334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.970300 5520 generic.go:356] "Generic (PLEG): container finished" podID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerID="383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217" exitCode=0 Oct 31 03:47:44.971750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.970340 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" Oct 31 03:47:44.971750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.970450 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" event={"ID":"c04bd3bb-7308-4d36-95d1-d35a1d8b431b","Type":"ContainerDied","Data":"383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217"} Oct 31 03:47:44.971750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.970626 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w" event={"ID":"c04bd3bb-7308-4d36-95d1-d35a1d8b431b","Type":"ContainerDied","Data":"afb9e6e6b06fab43ebe472d66b7cacb6ebb58c68081fc756e919a733075eea12"} Oct 31 03:47:44.971750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.970660 5520 scope.go:122] "RemoveContainer" containerID="383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217" Oct 31 03:47:44.984431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.984413 5520 scope.go:122] "RemoveContainer" containerID="383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217" Oct 31 03:47:44.984742 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:47:44.984705 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217\": container with ID starting with 383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217 not found: ID does not exist" containerID="383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217" Oct 31 03:47:44.984827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.984742 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217"} err="failed to get container status \"383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217\": rpc error: code = NotFound desc = could not find container \"383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217\": container with ID starting with 383fe8271861c03372fce49b4a4f4e73365e99ce6d2926218400e33b6b2f2217 not found: ID does not exist" Oct 31 03:47:45.000052 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:44.998902 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w"] Oct 31 03:47:45.004065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:45.004034 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-880e04/test-model-880e04-kserve-5ff89d8dbf-zfx4w"] Oct 31 03:47:45.877227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:45.875307 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" podUID="cc412090-34af-455d-b359-890af2e14f43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:46.015028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:46.014995 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" path="/var/lib/kubelet/pods/0694d1ee-0802-47de-a550-53f1da6e183b/volumes" Oct 31 03:47:46.068330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:46.067747 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" path="/var/lib/kubelet/pods/c04bd3bb-7308-4d36-95d1-d35a1d8b431b/volumes" Oct 31 03:47:46.905228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:46.904175 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" podUID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.129891 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql"] Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131490 5520 cpu_manager.go:414] "containerMap: removing container" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131513 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131539 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131547 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131618 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131628 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131669 5520 cpu_manager.go:414] "containerMap: removing container" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131676 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131688 5520 cpu_manager.go:414] "containerMap: removing container" podUID="aec13ba1-1656-4f86-99be-56aa932a2e2e" containerName="revoke-keys" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131696 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec13ba1-1656-4f86-99be-56aa932a2e2e" containerName="revoke-keys" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131706 5520 cpu_manager.go:414] "containerMap: removing container" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131713 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131888 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04bd3bb-7308-4d36-95d1-d35a1d8b431b" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131909 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e030301-62ab-4734-9bcf-cd9a0854817c" containerName="istio-proxy" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131918 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec13ba1-1656-4f86-99be-56aa932a2e2e" containerName="revoke-keys" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131926 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a35073-afca-4267-aa1b-4c8fe008f6b5" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131940 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="0694d1ee-0802-47de-a550-53f1da6e183b" containerName="main" Oct 31 03:47:47.133231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.131952 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3bfa65-ac7b-4b46-81d3-8319d1f54937" containerName="main" Oct 31 03:47:47.142279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.138325 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.142279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.138501 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql"] Oct 31 03:47:47.147716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.147687 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ce9b89-openshift-default-dockercfg-zsljk\"" Oct 31 03:47:47.274231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.273889 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxv2d\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-kube-api-access-hxv2d\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.273989 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-envoy\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.274080 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-credential-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.274117 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-certs\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.274286 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-token\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.274315 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-data\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.274414 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-podinfo\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.274490 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.274632 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.274557 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istiod-ca-cert\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.617775 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-certs\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.617988 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-token\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.618026 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-data\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.618148 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-podinfo\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.618285 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.618390 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istiod-ca-cert\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.618499 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxv2d\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-kube-api-access-hxv2d\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.618598 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-envoy\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.618713 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-credential-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.620417 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-certs\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.621103 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istiod-ca-cert\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.621408 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.623578 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-data\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.625811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.624946 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-credential-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.672238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.671806 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-envoy\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.682696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.676833 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-podinfo\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.695251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.692570 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-token\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.723739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.722528 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxv2d\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-kube-api-access-hxv2d\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:47.771686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:47.771644 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:48.207017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:48.206993 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql"] Oct 31 03:47:48.209675 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:47:48.209645 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aec880_cd9a_46fb_99a6_4a51e8921c94.slice/crio-3447455847a126c026a523c582344ab174c6d95caa00837e01dd955574d3ec47 WatchSource:0}: Error finding container 3447455847a126c026a523c582344ab174c6d95caa00837e01dd955574d3ec47: Status 404 returned error can't find the container with id 3447455847a126c026a523c582344ab174c6d95caa00837e01dd955574d3ec47 Oct 31 03:47:48.213475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:48.213436 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:47:48.213572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:48.213527 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:47:48.213618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:48.213571 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:47:48.626292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:48.625604 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll"] Oct 31 03:47:48.634322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:48.634269 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6"] Oct 31 03:47:49.463984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:49.463941 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" event={"ID":"f2aec880-cd9a-46fb-99a6-4a51e8921c94","Type":"ContainerStarted","Data":"6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7"} Oct 31 03:47:49.464656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:49.464629 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" event={"ID":"f2aec880-cd9a-46fb-99a6-4a51e8921c94","Type":"ContainerStarted","Data":"3447455847a126c026a523c582344ab174c6d95caa00837e01dd955574d3ec47"} Oct 31 03:47:49.772610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:49.772522 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:49.780712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:49.780679 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:49.807409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:49.807343 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podStartSLOduration=2.807322582 podStartE2EDuration="2.807322582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:49.516154979 +0000 UTC m=+1893.710678117" watchObservedRunningTime="2026-10-31 03:47:49.807322582 +0000 UTC m=+1894.001845704" Oct 31 03:47:50.473315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:50.473283 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:50.474313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:50.474291 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:47:51.801040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:51.796834 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b"] Oct 31 03:47:51.803065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:51.803042 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2dc82-2j22b"] Oct 31 03:47:51.930316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:51.930279 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec13ba1-1656-4f86-99be-56aa932a2e2e" path="/var/lib/kubelet/pods/aec13ba1-1656-4f86-99be-56aa932a2e2e/volumes" Oct 31 03:47:53.638235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:53.635968 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" podUID="23090f24-1e03-4f47-8fc6-3ccdaae08883" containerName="payload-pre-processing" containerID="cri-o://fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd" gracePeriod=25 Oct 31 03:47:53.638235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:53.636166 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" podUID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" containerName="payload-processing" containerID="cri-o://2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3" gracePeriod=25 Oct 31 03:47:55.849125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:55.849027 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" podUID="cc412090-34af-455d-b359-890af2e14f43" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:56.771582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:56.771540 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" podUID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:56.772927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:56.772895 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" podUID="23090f24-1e03-4f47-8fc6-3ccdaae08883" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:56.854599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:56.854566 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" podUID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:47:59.515962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:59.515919 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:47:59.550299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:47:59.550247 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:48:04.334382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.333972 5520 generic.go:356] "Generic (PLEG): container finished" podID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" containerID="f6b5bfa763e3c426cc7125e888b4a90f0d79eac5eb3ed5a37ef98c305be9912e" exitCode=0 Oct 31 03:48:04.334382 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.334251 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" event={"ID":"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b","Type":"ContainerDied","Data":"f6b5bfa763e3c426cc7125e888b4a90f0d79eac5eb3ed5a37ef98c305be9912e"} Oct 31 03:48:04.337868 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.337846 5520 generic.go:356] "Generic (PLEG): container finished" podID="cc412090-34af-455d-b359-890af2e14f43" containerID="9a4c6560473e28db8b728935ab306f48cf14cd8f734a72d823b9f8a6afed684e" exitCode=0 Oct 31 03:48:04.338078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.338063 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" event={"ID":"cc412090-34af-455d-b359-890af2e14f43","Type":"ContainerDied","Data":"9a4c6560473e28db8b728935ab306f48cf14cd8f734a72d823b9f8a6afed684e"} Oct 31 03:48:04.379889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.379855 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:48:04.481563 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.481530 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:48:04.526754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.516043 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc412090-34af-455d-b359-890af2e14f43-kube-api-access-tbpbs\" (UniqueName: \"kubernetes.io/projected/cc412090-34af-455d-b359-890af2e14f43-kube-api-access-tbpbs\") pod \"cc412090-34af-455d-b359-890af2e14f43\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " Oct 31 03:48:04.526754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.516357 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc412090-34af-455d-b359-890af2e14f43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc412090-34af-455d-b359-890af2e14f43-plugins-config-volume\") pod \"cc412090-34af-455d-b359-890af2e14f43\" (UID: \"cc412090-34af-455d-b359-890af2e14f43\") " Oct 31 03:48:04.526754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.523161 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc412090-34af-455d-b359-890af2e14f43-plugins-config-volume" pod "cc412090-34af-455d-b359-890af2e14f43" (UID: "cc412090-34af-455d-b359-890af2e14f43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:04.551913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.551841 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc412090-34af-455d-b359-890af2e14f43-kube-api-access-tbpbs" pod "cc412090-34af-455d-b359-890af2e14f43" (UID: "cc412090-34af-455d-b359-890af2e14f43"). InnerVolumeSpecName "kube-api-access-tbpbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:04.618241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.618189 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-kube-api-access-rk8fg\" (UniqueName: \"kubernetes.io/projected/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-kube-api-access-rk8fg\") pod \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " Oct 31 03:48:04.618529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.618311 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-plugins-config-volume\") pod \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\" (UID: \"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b\") " Oct 31 03:48:04.618758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.618731 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-plugins-config-volume" pod "53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" (UID: "53dfd6d1-27c7-4e50-85ae-e1807ff4b76b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:04.619061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.619039 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbpbs\" (UniqueName: \"kubernetes.io/projected/cc412090-34af-455d-b359-890af2e14f43-kube-api-access-tbpbs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.619128 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.619069 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.619128 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.619090 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc412090-34af-455d-b359-890af2e14f43-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.620545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.620511 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-kube-api-access-rk8fg" pod "53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" (UID: "53dfd6d1-27c7-4e50-85ae-e1807ff4b76b"). InnerVolumeSpecName "kube-api-access-rk8fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:04.720922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:04.720879 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8fg\" (UniqueName: \"kubernetes.io/projected/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b-kube-api-access-rk8fg\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.348955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.348866 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" Oct 31 03:48:05.348955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.348909 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk" event={"ID":"53dfd6d1-27c7-4e50-85ae-e1807ff4b76b","Type":"ContainerDied","Data":"fc035a18948804f7fc946592fc1cb1b76bcbcc50e204706af45ed00eb00596a7"} Oct 31 03:48:05.349547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.348971 5520 scope.go:122] "RemoveContainer" containerID="f6b5bfa763e3c426cc7125e888b4a90f0d79eac5eb3ed5a37ef98c305be9912e" Oct 31 03:48:05.351156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.351103 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" event={"ID":"cc412090-34af-455d-b359-890af2e14f43","Type":"ContainerDied","Data":"de1b9d391e583c23ed4a21b75124be7009f9ec061d6282c555f302ddcbcdedc4"} Oct 31 03:48:05.351401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.351377 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q" Oct 31 03:48:05.370946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.370821 5520 scope.go:122] "RemoveContainer" containerID="9a4c6560473e28db8b728935ab306f48cf14cd8f734a72d823b9f8a6afed684e" Oct 31 03:48:05.382264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.382232 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk"] Oct 31 03:48:05.389911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.389879 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dc82-589b579679-kvnfk"] Oct 31 03:48:05.416248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.415284 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q"] Oct 31 03:48:05.421055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.421028 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dc82-758456556c-mjp8q"] Oct 31 03:48:05.923841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.923811 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" path="/var/lib/kubelet/pods/53dfd6d1-27c7-4e50-85ae-e1807ff4b76b/volumes" Oct 31 03:48:05.924436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:05.924420 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc412090-34af-455d-b359-890af2e14f43" path="/var/lib/kubelet/pods/cc412090-34af-455d-b359-890af2e14f43/volumes" Oct 31 03:48:06.773322 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:06.773290 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" podUID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:48:06.773990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:06.773966 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" podUID="23090f24-1e03-4f47-8fc6-3ccdaae08883" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:48:07.280766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:07.280439 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk"] Oct 31 03:48:07.280766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:07.280703 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" containerID="cri-o://918a85cdb5fc16a63b8f354a83e0085ef3f417d66e443252d1b41c46171882a7" gracePeriod=30 Oct 31 03:48:09.362581 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:09.362540 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:48:09.447420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:09.447380 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l"] Oct 31 03:48:09.456911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:09.456879 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8"] Oct 31 03:48:12.528351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.528123 5520 generic.go:356] "Generic (PLEG): container finished" podID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerID="918a85cdb5fc16a63b8f354a83e0085ef3f417d66e443252d1b41c46171882a7" exitCode=0 Oct 31 03:48:12.528351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.528151 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" event={"ID":"72581b3c-56db-4876-b4e2-2feddb1126c0","Type":"ContainerDied","Data":"918a85cdb5fc16a63b8f354a83e0085ef3f417d66e443252d1b41c46171882a7"} Oct 31 03:48:12.643804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.643768 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:48:12.736637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736600 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-certs\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.736801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736716 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-data\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.736801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736769 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-token\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-token\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.736929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736812 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-podinfo\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.736929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736839 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72581b3c-56db-4876-b4e2-2feddb1126c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72581b3c-56db-4876-b4e2-2feddb1126c0-istiod-ca-cert\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.736929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736866 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-socket\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.737109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736943 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-data" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.737109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.736917 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-certs" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.737237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.737105 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-envoy\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.737237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.737150 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-credential-socket\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.737374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.737320 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-kube-api-access-9r8mq\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-kube-api-access-9r8mq\") pod \"72581b3c-56db-4876-b4e2-2feddb1126c0\" (UID: \"72581b3c-56db-4876-b4e2-2feddb1126c0\") " Oct 31 03:48:12.737506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.737481 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-credential-socket" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.737585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.737510 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72581b3c-56db-4876-b4e2-2feddb1126c0-istiod-ca-cert" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:12.737654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.737584 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-socket" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.738498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.738471 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.738622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.738501 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.738622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.738517 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.738622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.738531 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72581b3c-56db-4876-b4e2-2feddb1126c0-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.738622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.738545 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.739674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.739642 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-token" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.740887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.740848 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-podinfo" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:12.740887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.740854 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-envoy" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.741275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.741174 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-kube-api-access-9r8mq" pod "72581b3c-56db-4876-b4e2-2feddb1126c0" (UID: "72581b3c-56db-4876-b4e2-2feddb1126c0"). InnerVolumeSpecName "kube-api-access-9r8mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.840817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.840721 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.840817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.840763 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.840817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.840772 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72581b3c-56db-4876-b4e2-2feddb1126c0-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.840817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:12.840781 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8mq\" (UniqueName: \"kubernetes.io/projected/72581b3c-56db-4876-b4e2-2feddb1126c0-kube-api-access-9r8mq\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:13.538347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:13.538313 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" Oct 31 03:48:13.538828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:13.538314 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk" event={"ID":"72581b3c-56db-4876-b4e2-2feddb1126c0","Type":"ContainerDied","Data":"14fd71052f6cb0cfc7db0781d043fbf5e5c08e3db16679d6382a53f740d530b1"} Oct 31 03:48:13.538828 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:13.538462 5520 scope.go:122] "RemoveContainer" containerID="918a85cdb5fc16a63b8f354a83e0085ef3f417d66e443252d1b41c46171882a7" Oct 31 03:48:13.564418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:13.564384 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk"] Oct 31 03:48:13.571468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:13.571437 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-czblk"] Oct 31 03:48:13.920663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:13.920635 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:48:13.928646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:13.928617 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" path="/var/lib/kubelet/pods/72581b3c-56db-4876-b4e2-2feddb1126c0/volumes" Oct 31 03:48:14.016408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.007159 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a993c9e-f98b-4d94-81ee-6c51e6eda148-kube-api-access-fffm6\" (UniqueName: \"kubernetes.io/projected/5a993c9e-f98b-4d94-81ee-6c51e6eda148-kube-api-access-fffm6\") pod \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " Oct 31 03:48:14.016408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.007320 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a993c9e-f98b-4d94-81ee-6c51e6eda148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a993c9e-f98b-4d94-81ee-6c51e6eda148-plugins-config-volume\") pod \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\" (UID: \"5a993c9e-f98b-4d94-81ee-6c51e6eda148\") " Oct 31 03:48:14.058242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.017042 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a993c9e-f98b-4d94-81ee-6c51e6eda148-plugins-config-volume" pod "5a993c9e-f98b-4d94-81ee-6c51e6eda148" (UID: "5a993c9e-f98b-4d94-81ee-6c51e6eda148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:14.058242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.021843 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a993c9e-f98b-4d94-81ee-6c51e6eda148-kube-api-access-fffm6" pod "5a993c9e-f98b-4d94-81ee-6c51e6eda148" (UID: "5a993c9e-f98b-4d94-81ee-6c51e6eda148"). InnerVolumeSpecName "kube-api-access-fffm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:14.136477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.122054 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fffm6\" (UniqueName: \"kubernetes.io/projected/5a993c9e-f98b-4d94-81ee-6c51e6eda148-kube-api-access-fffm6\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.217065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.217042 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a993c9e-f98b-4d94-81ee-6c51e6eda148-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.235183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.234824 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:48:14.354488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.353247 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23090f24-1e03-4f47-8fc6-3ccdaae08883-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23090f24-1e03-4f47-8fc6-3ccdaae08883-plugins-config-volume\") pod \"23090f24-1e03-4f47-8fc6-3ccdaae08883\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " Oct 31 03:48:14.354488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.353329 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23090f24-1e03-4f47-8fc6-3ccdaae08883-kube-api-access-mpn56\" (UniqueName: \"kubernetes.io/projected/23090f24-1e03-4f47-8fc6-3ccdaae08883-kube-api-access-mpn56\") pod \"23090f24-1e03-4f47-8fc6-3ccdaae08883\" (UID: \"23090f24-1e03-4f47-8fc6-3ccdaae08883\") " Oct 31 03:48:14.368684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.368638 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23090f24-1e03-4f47-8fc6-3ccdaae08883-plugins-config-volume" pod "23090f24-1e03-4f47-8fc6-3ccdaae08883" (UID: "23090f24-1e03-4f47-8fc6-3ccdaae08883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:14.416748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.416504 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23090f24-1e03-4f47-8fc6-3ccdaae08883-kube-api-access-mpn56" pod "23090f24-1e03-4f47-8fc6-3ccdaae08883" (UID: "23090f24-1e03-4f47-8fc6-3ccdaae08883"). InnerVolumeSpecName "kube-api-access-mpn56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:14.447944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.447881 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" podUID="6fc6d6f3-7ba7-4e5d-9e97-219354a72025" containerName="payload-processing" containerID="cri-o://4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f" gracePeriod=25 Oct 31 03:48:14.455961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.455936 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23090f24-1e03-4f47-8fc6-3ccdaae08883-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.456064 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.455968 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpn56\" (UniqueName: \"kubernetes.io/projected/23090f24-1e03-4f47-8fc6-3ccdaae08883-kube-api-access-mpn56\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.457432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.457401 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" podUID="004fe8fb-5074-440f-b43c-f95f17fdbba0" containerName="payload-pre-processing" containerID="cri-o://33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821" gracePeriod=25 Oct 31 03:48:14.552116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.551387 5520 generic.go:356] "Generic (PLEG): container finished" podID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" containerID="2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3" exitCode=0 Oct 31 03:48:14.552116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.551465 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" Oct 31 03:48:14.552116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.551628 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" event={"ID":"5a993c9e-f98b-4d94-81ee-6c51e6eda148","Type":"ContainerDied","Data":"2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3"} Oct 31 03:48:14.552116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.551675 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll" event={"ID":"5a993c9e-f98b-4d94-81ee-6c51e6eda148","Type":"ContainerDied","Data":"623efa4d33dc01f89bc5cfefba4248b6c47e7a8cde16e0cfb593f1ccb0ef6b00"} Oct 31 03:48:14.552116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.551705 5520 scope.go:122] "RemoveContainer" containerID="2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3" Oct 31 03:48:14.558737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.558711 5520 generic.go:356] "Generic (PLEG): container finished" podID="23090f24-1e03-4f47-8fc6-3ccdaae08883" containerID="fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd" exitCode=0 Oct 31 03:48:14.558881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.558777 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" Oct 31 03:48:14.558881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.558778 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" event={"ID":"23090f24-1e03-4f47-8fc6-3ccdaae08883","Type":"ContainerDied","Data":"fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd"} Oct 31 03:48:14.558881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.558874 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6" event={"ID":"23090f24-1e03-4f47-8fc6-3ccdaae08883","Type":"ContainerDied","Data":"a46c91f487b373ee382271a9569bf13219cfb940a360fa4e650d1e45ca0677d9"} Oct 31 03:48:14.575547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.575433 5520 scope.go:122] "RemoveContainer" containerID="2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3" Oct 31 03:48:14.575818 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:14.575789 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3\": container with ID starting with 2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3 not found: ID does not exist" containerID="2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3" Oct 31 03:48:14.576510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.575828 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3"} err="failed to get container status \"2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3\": rpc error: code = NotFound desc = could not find container \"2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3\": container with ID starting with 2c79c05b663f54bc576477b5a9878f2cf9b9b0d4e101bc7f2ec91d9bc9e8bfb3 not found: ID does not exist" Oct 31 03:48:14.576510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.575853 5520 scope.go:122] "RemoveContainer" containerID="fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd" Oct 31 03:48:14.598392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.598354 5520 scope.go:122] "RemoveContainer" containerID="fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd" Oct 31 03:48:14.602639 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:14.602605 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd\": container with ID starting with fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd not found: ID does not exist" containerID="fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd" Oct 31 03:48:14.602808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.602780 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd"} err="failed to get container status \"fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd\": rpc error: code = NotFound desc = could not find container \"fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd\": container with ID starting with fc5ffd1204f6d6eeeafc129701afcdcdf20c3e20049f001e1d828c89eed1a7bd not found: ID does not exist" Oct 31 03:48:14.623717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.623688 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll"] Oct 31 03:48:14.633035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.632989 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c5dcb7-7cd6b97849-v48ll"] Oct 31 03:48:14.637602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.637570 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6"] Oct 31 03:48:14.642518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.642160 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c5dcb7-5d4d97f99f-zfzn6"] Oct 31 03:48:14.784566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.784535 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:48:14.855300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.855273 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:48:14.863329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.863301 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004fe8fb-5074-440f-b43c-f95f17fdbba0-kube-api-access-8kzgq\" (UniqueName: \"kubernetes.io/projected/004fe8fb-5074-440f-b43c-f95f17fdbba0-kube-api-access-8kzgq\") pod \"004fe8fb-5074-440f-b43c-f95f17fdbba0\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " Oct 31 03:48:14.863591 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.863571 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-plugins-config-volume\") pod \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " Oct 31 03:48:14.863671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.863656 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004fe8fb-5074-440f-b43c-f95f17fdbba0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004fe8fb-5074-440f-b43c-f95f17fdbba0-plugins-config-volume\") pod \"004fe8fb-5074-440f-b43c-f95f17fdbba0\" (UID: \"004fe8fb-5074-440f-b43c-f95f17fdbba0\") " Oct 31 03:48:14.863748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.863730 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-kube-api-access-9gfgz\" (UniqueName: \"kubernetes.io/projected/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-kube-api-access-9gfgz\") pod \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\" (UID: \"6fc6d6f3-7ba7-4e5d-9e97-219354a72025\") " Oct 31 03:48:14.864013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.863993 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004fe8fb-5074-440f-b43c-f95f17fdbba0-plugins-config-volume" pod "004fe8fb-5074-440f-b43c-f95f17fdbba0" (UID: "004fe8fb-5074-440f-b43c-f95f17fdbba0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:14.864076 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.863994 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-plugins-config-volume" pod "6fc6d6f3-7ba7-4e5d-9e97-219354a72025" (UID: "6fc6d6f3-7ba7-4e5d-9e97-219354a72025"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:14.864450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.864431 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.864524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.864453 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004fe8fb-5074-440f-b43c-f95f17fdbba0-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.865786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.865768 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004fe8fb-5074-440f-b43c-f95f17fdbba0-kube-api-access-8kzgq" pod "004fe8fb-5074-440f-b43c-f95f17fdbba0" (UID: "004fe8fb-5074-440f-b43c-f95f17fdbba0"). InnerVolumeSpecName "kube-api-access-8kzgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:14.865920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.865899 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-kube-api-access-9gfgz" pod "6fc6d6f3-7ba7-4e5d-9e97-219354a72025" (UID: "6fc6d6f3-7ba7-4e5d-9e97-219354a72025"). InnerVolumeSpecName "kube-api-access-9gfgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:14.979528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.979485 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gfgz\" (UniqueName: \"kubernetes.io/projected/6fc6d6f3-7ba7-4e5d-9e97-219354a72025-kube-api-access-9gfgz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:14.979528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:14.979522 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzgq\" (UniqueName: \"kubernetes.io/projected/004fe8fb-5074-440f-b43c-f95f17fdbba0-kube-api-access-8kzgq\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:15.571342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.571296 5520 generic.go:356] "Generic (PLEG): container finished" podID="6fc6d6f3-7ba7-4e5d-9e97-219354a72025" containerID="4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f" exitCode=0 Oct 31 03:48:15.571834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.571361 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" Oct 31 03:48:15.571834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.571400 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" event={"ID":"6fc6d6f3-7ba7-4e5d-9e97-219354a72025","Type":"ContainerDied","Data":"4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f"} Oct 31 03:48:15.571834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.571469 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l" event={"ID":"6fc6d6f3-7ba7-4e5d-9e97-219354a72025","Type":"ContainerDied","Data":"29512d8a7915a7e54476e90cdd4cd7adc3cb38e946465567b80c2c97697115b2"} Oct 31 03:48:15.571834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.571498 5520 scope.go:122] "RemoveContainer" containerID="4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f" Oct 31 03:48:15.576121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.575834 5520 generic.go:356] "Generic (PLEG): container finished" podID="004fe8fb-5074-440f-b43c-f95f17fdbba0" containerID="33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821" exitCode=0 Oct 31 03:48:15.576121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.575938 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" event={"ID":"004fe8fb-5074-440f-b43c-f95f17fdbba0","Type":"ContainerDied","Data":"33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821"} Oct 31 03:48:15.576121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.575979 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" event={"ID":"004fe8fb-5074-440f-b43c-f95f17fdbba0","Type":"ContainerDied","Data":"6be7574bc26b3def790fc10c89739810cc97f2e9cd9638323e613b30d4bef4d9"} Oct 31 03:48:15.576121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.576003 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8" Oct 31 03:48:15.594149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.594111 5520 scope.go:122] "RemoveContainer" containerID="4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f" Oct 31 03:48:15.594629 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:15.594600 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f\": container with ID starting with 4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f not found: ID does not exist" containerID="4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f" Oct 31 03:48:15.594703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.594642 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f"} err="failed to get container status \"4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f\": rpc error: code = NotFound desc = could not find container \"4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f\": container with ID starting with 4eaaf77ab7392be03a84b39d297aea5f75f18bd97745507b21a12fc662972f9f not found: ID does not exist" Oct 31 03:48:15.594703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.594666 5520 scope.go:122] "RemoveContainer" containerID="33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821" Oct 31 03:48:15.612578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.612546 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l"] Oct 31 03:48:15.612879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.612860 5520 scope.go:122] "RemoveContainer" containerID="33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821" Oct 31 03:48:15.613427 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:15.613404 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821\": container with ID starting with 33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821 not found: ID does not exist" containerID="33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821" Oct 31 03:48:15.613548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.613434 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821"} err="failed to get container status \"33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821\": rpc error: code = NotFound desc = could not find container \"33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821\": container with ID starting with 33f23536f7277e63ce28bd1cb93e85d1b06c9c25cfac7f0b227cdb208fca6821 not found: ID does not exist" Oct 31 03:48:15.620337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.620316 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-880e04-59df8799c7-qxn4l"] Oct 31 03:48:15.628247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.628187 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8"] Oct 31 03:48:15.632721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.632700 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-880e04-64c97f5bf5-jh2h8"] Oct 31 03:48:15.995238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:15.995113 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004fe8fb-5074-440f-b43c-f95f17fdbba0" path="/var/lib/kubelet/pods/004fe8fb-5074-440f-b43c-f95f17fdbba0/volumes" Oct 31 03:48:16.013227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:16.010946 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23090f24-1e03-4f47-8fc6-3ccdaae08883" path="/var/lib/kubelet/pods/23090f24-1e03-4f47-8fc6-3ccdaae08883/volumes" Oct 31 03:48:16.013227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:16.011646 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" path="/var/lib/kubelet/pods/5a993c9e-f98b-4d94-81ee-6c51e6eda148/volumes" Oct 31 03:48:16.013227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:16.012156 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc6d6f3-7ba7-4e5d-9e97-219354a72025" path="/var/lib/kubelet/pods/6fc6d6f3-7ba7-4e5d-9e97-219354a72025/volumes" Oct 31 03:48:26.365974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:26.365931 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt"] Oct 31 03:48:31.366941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.366899 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" podUID="9101743e-f614-4906-bd37-af2d2e785874" containerName="payload-processing" containerID="cri-o://6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9" gracePeriod=25 Oct 31 03:48:31.745233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.724456 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:48:31.818040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.818001 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9101743e-f614-4906-bd37-af2d2e785874-kube-api-access-wh85b\" (UniqueName: \"kubernetes.io/projected/9101743e-f614-4906-bd37-af2d2e785874-kube-api-access-wh85b\") pod \"9101743e-f614-4906-bd37-af2d2e785874\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " Oct 31 03:48:31.818317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.818106 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9101743e-f614-4906-bd37-af2d2e785874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9101743e-f614-4906-bd37-af2d2e785874-plugins-config-volume\") pod \"9101743e-f614-4906-bd37-af2d2e785874\" (UID: \"9101743e-f614-4906-bd37-af2d2e785874\") " Oct 31 03:48:31.818623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.818590 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9101743e-f614-4906-bd37-af2d2e785874-plugins-config-volume" pod "9101743e-f614-4906-bd37-af2d2e785874" (UID: "9101743e-f614-4906-bd37-af2d2e785874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:31.819480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.819453 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9101743e-f614-4906-bd37-af2d2e785874-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:31.820820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.820795 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9101743e-f614-4906-bd37-af2d2e785874-kube-api-access-wh85b" pod "9101743e-f614-4906-bd37-af2d2e785874" (UID: "9101743e-f614-4906-bd37-af2d2e785874"). InnerVolumeSpecName "kube-api-access-wh85b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:31.926865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:31.926764 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh85b\" (UniqueName: \"kubernetes.io/projected/9101743e-f614-4906-bd37-af2d2e785874-kube-api-access-wh85b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:32.099483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.099448 5520 generic.go:356] "Generic (PLEG): container finished" podID="9101743e-f614-4906-bd37-af2d2e785874" containerID="6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9" exitCode=0 Oct 31 03:48:32.099700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.099534 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" Oct 31 03:48:32.099700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.099537 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" event={"ID":"9101743e-f614-4906-bd37-af2d2e785874","Type":"ContainerDied","Data":"6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9"} Oct 31 03:48:32.099700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.099606 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt" event={"ID":"9101743e-f614-4906-bd37-af2d2e785874","Type":"ContainerDied","Data":"c0e596880cd8368f6550d7e77bd935a41e0b909899116dffbfb15784856561a8"} Oct 31 03:48:32.099700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.099635 5520 scope.go:122] "RemoveContainer" containerID="6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9" Oct 31 03:48:32.114295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.114262 5520 scope.go:122] "RemoveContainer" containerID="6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9" Oct 31 03:48:32.115159 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:32.115134 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9\": container with ID starting with 6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9 not found: ID does not exist" containerID="6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9" Oct 31 03:48:32.115306 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.115167 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9"} err="failed to get container status \"6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9\": rpc error: code = NotFound desc = could not find container \"6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9\": container with ID starting with 6d8377714e5aa6b6db704293702b662fa6ef5a07afb3beb60ac121010670cca9 not found: ID does not exist" Oct 31 03:48:32.119469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.119442 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt"] Oct 31 03:48:32.123834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.123810 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-73e222-64859d8b69-n7ftt"] Oct 31 03:48:32.498004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.497946 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh"] Oct 31 03:48:32.498839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:32.498805 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" containerID="cri-o://d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870" gracePeriod=30 Oct 31 03:48:33.896863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:33.896822 5520 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:48:33.897413 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:33.896905 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:48:33.924906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:33.924870 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9101743e-f614-4906-bd37-af2d2e785874" path="/var/lib/kubelet/pods/9101743e-f614-4906-bd37-af2d2e785874/volumes" Oct 31 03:48:34.064870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.064832 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2"] Oct 31 03:48:34.066131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066104 5520 cpu_manager.go:414] "containerMap: removing container" podUID="23090f24-1e03-4f47-8fc6-3ccdaae08883" containerName="payload-pre-processing" Oct 31 03:48:34.066389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066360 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23090f24-1e03-4f47-8fc6-3ccdaae08883" containerName="payload-pre-processing" Oct 31 03:48:34.066389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066392 5520 cpu_manager.go:414] "containerMap: removing container" podUID="004fe8fb-5074-440f-b43c-f95f17fdbba0" containerName="payload-pre-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066402 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004fe8fb-5074-440f-b43c-f95f17fdbba0" containerName="payload-pre-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066434 5520 cpu_manager.go:414] "containerMap: removing container" podUID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" containerName="payload-pre-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066443 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" containerName="payload-pre-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066494 5520 cpu_manager.go:414] "containerMap: removing container" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066504 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066522 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" containerName="payload-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066530 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" containerName="payload-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066546 5520 cpu_manager.go:414] "containerMap: removing container" podUID="cc412090-34af-455d-b359-890af2e14f43" containerName="payload-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066553 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc412090-34af-455d-b359-890af2e14f43" containerName="payload-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066578 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9101743e-f614-4906-bd37-af2d2e785874" containerName="payload-processing" Oct 31 03:48:34.066590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066585 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9101743e-f614-4906-bd37-af2d2e785874" containerName="payload-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066615 5520 cpu_manager.go:414] "containerMap: removing container" podUID="6fc6d6f3-7ba7-4e5d-9e97-219354a72025" containerName="payload-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066622 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc6d6f3-7ba7-4e5d-9e97-219354a72025" containerName="payload-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066848 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="23090f24-1e03-4f47-8fc6-3ccdaae08883" containerName="payload-pre-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066868 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="004fe8fb-5074-440f-b43c-f95f17fdbba0" containerName="payload-pre-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066887 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dfd6d1-27c7-4e50-85ae-e1807ff4b76b" containerName="payload-pre-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066902 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="72581b3c-56db-4876-b4e2-2feddb1126c0" containerName="istio-proxy" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066911 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a993c9e-f98b-4d94-81ee-6c51e6eda148" containerName="payload-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066922 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc412090-34af-455d-b359-890af2e14f43" containerName="payload-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066932 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9101743e-f614-4906-bd37-af2d2e785874" containerName="payload-processing" Oct 31 03:48:34.067265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.066970 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc6d6f3-7ba7-4e5d-9e97-219354a72025" containerName="payload-processing" Oct 31 03:48:34.091810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.091780 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2"] Oct 31 03:48:34.092150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.092106 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.099920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.099892 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-afb8be\"/\"openshift-service-ca.crt\"" Oct 31 03:48:34.100261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.100241 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-afb8be\"/\"default-dockercfg-8c6qw\"" Oct 31 03:48:34.100486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.100469 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-afb8be\"/\"auth-test-model-afb8be-kserve-self-signed-certs\"" Oct 31 03:48:34.100716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.100700 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-afb8be\"/\"kube-root-ca.crt\"" Oct 31 03:48:34.204098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.204062 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-dshm\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.204345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.204171 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-home\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.204589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.204559 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh69s\" (UniqueName: \"kubernetes.io/projected/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-kube-api-access-qh69s\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.204852 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.204815 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tmp-dir\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.204930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.204895 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-model-cache\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.205010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.204994 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tls-certs\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.307830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.307785 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tls-certs\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.309166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.309139 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-dshm\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.309449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.309429 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-home\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.310131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.310109 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh69s\" (UniqueName: \"kubernetes.io/projected/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-kube-api-access-qh69s\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.310302 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.310286 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tmp-dir\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.310460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.310444 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-model-cache\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.318226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.311055 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-model-cache\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.318424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.312583 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-home\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.318529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.313647 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tmp-dir\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.364227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.325095 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tls-certs\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.364227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.334255 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh69s\" (UniqueName: \"kubernetes.io/projected/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-kube-api-access-qh69s\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.365225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.365178 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-dshm\") pod \"auth-test-model-afb8be-kserve-5fd464f665-p24m2\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:34.429643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:34.429606 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:35.108115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:35.108080 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2"] Oct 31 03:48:35.205692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:35.205658 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" event={"ID":"ab4f5aae-be3b-4e3b-b0df-a66c622e950a","Type":"ContainerStarted","Data":"a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942"} Oct 31 03:48:35.205886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:35.205705 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" event={"ID":"ab4f5aae-be3b-4e3b-b0df-a66c622e950a","Type":"ContainerStarted","Data":"4a5ac083e50cc3f906671d2b66f5e50b64cd8d043d79fa621f0e275d36f4671b"} Oct 31 03:48:35.227696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:35.227643 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" podStartSLOduration=1.227630031 podStartE2EDuration="1.227630031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:35.223226471 +0000 UTC m=+1939.417749620" watchObservedRunningTime="2026-10-31 03:48:35.227630031 +0000 UTC m=+1939.422153151" Oct 31 03:48:35.439407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:35.432309 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:37.941445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.941416 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:48:37.982041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.982005 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-credential-socket\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.982386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.982360 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-token\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-token\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.982547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.982529 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-podinfo\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.982781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.982728 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istiod-ca-cert\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.982955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.982937 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-data\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.983150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.983133 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-socket\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.983352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.983333 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-kube-api-access-wzv8x\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-kube-api-access-wzv8x\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.983755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.983734 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-certs\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.983902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.983885 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-envoy\") pod \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\" (UID: \"929fc3a7-eaf6-4d34-8754-dec95c4aff02\") " Oct 31 03:48:37.985738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.985715 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-socket" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.986345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.985996 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istiod-ca-cert" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:37.986464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.986444 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-certs" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.986587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.986338 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-data" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.986587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:37.986393 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-credential-socket" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:38.008699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.008648 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-envoy" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:38.023292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.020734 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-podinfo" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:38.023292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.021754 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-token" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:38.023292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.022259 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-kube-api-access-wzv8x" pod "929fc3a7-eaf6-4d34-8754-dec95c4aff02" (UID: "929fc3a7-eaf6-4d34-8754-dec95c4aff02"). InnerVolumeSpecName "kube-api-access-wzv8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:38.087049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087016 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087050 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087062 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087082 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087096 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087116 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087129 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzv8x\" (UniqueName: \"kubernetes.io/projected/929fc3a7-eaf6-4d34-8754-dec95c4aff02-kube-api-access-wzv8x\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087143 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.087315 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.087154 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929fc3a7-eaf6-4d34-8754-dec95c4aff02-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.243281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.243242 5520 generic.go:356] "Generic (PLEG): container finished" podID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerID="d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870" exitCode=0 Oct 31 03:48:38.243508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.243310 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" Oct 31 03:48:38.243508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.243331 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" event={"ID":"929fc3a7-eaf6-4d34-8754-dec95c4aff02","Type":"ContainerDied","Data":"d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870"} Oct 31 03:48:38.243508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.243404 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh" event={"ID":"929fc3a7-eaf6-4d34-8754-dec95c4aff02","Type":"ContainerDied","Data":"bfe757b5bb638da9bb756b19bf868b48667026d2b585370a8b16997cbcf8798b"} Oct 31 03:48:38.243508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.243439 5520 scope.go:122] "RemoveContainer" containerID="d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870" Oct 31 03:48:38.260408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.260381 5520 scope.go:122] "RemoveContainer" containerID="a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7" Oct 31 03:48:38.274179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.274099 5520 scope.go:122] "RemoveContainer" containerID="d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870" Oct 31 03:48:38.274488 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:38.274460 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870\": container with ID starting with d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870 not found: ID does not exist" containerID="d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870" Oct 31 03:48:38.274620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.274490 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870"} err="failed to get container status \"d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870\": rpc error: code = NotFound desc = could not find container \"d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870\": container with ID starting with d1a780d71751ba2ff4b9ffc83a9ae6258280ef7e5ced5d6090d72d6815909870 not found: ID does not exist" Oct 31 03:48:38.274620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.274508 5520 scope.go:122] "RemoveContainer" containerID="a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7" Oct 31 03:48:38.274821 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:38.274800 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7\": container with ID starting with a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7 not found: ID does not exist" containerID="a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7" Oct 31 03:48:38.274882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.274825 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7"} err="failed to get container status \"a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7\": rpc error: code = NotFound desc = could not find container \"a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7\": container with ID starting with a6939a1a7b048c2098c6a5b6695b4ee4ef64f7535ce21e633e88d4fea95c89e7 not found: ID does not exist" Oct 31 03:48:38.282872 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.282840 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh"] Oct 31 03:48:38.289405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:38.289370 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-7plsh"] Oct 31 03:48:39.363977 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:39.363937 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:48:39.925256 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:39.925190 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" path="/var/lib/kubelet/pods/929fc3a7-eaf6-4d34-8754-dec95c4aff02/volumes" Oct 31 03:48:44.430175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:44.430142 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:44.444217 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:44.444182 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:45.443393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:45.443364 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:48:49.878671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:49.878509 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7"] Oct 31 03:48:49.879196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:49.878781 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" containerID="cri-o://78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c" gracePeriod=30 Oct 31 03:48:53.757796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.757756 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254"] Oct 31 03:48:53.759088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.759055 5520 cpu_manager.go:414] "containerMap: removing container" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" Oct 31 03:48:53.759088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.759086 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" Oct 31 03:48:53.759341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.759138 5520 cpu_manager.go:414] "containerMap: removing container" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" Oct 31 03:48:53.759341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.759146 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" Oct 31 03:48:53.759460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.759354 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" Oct 31 03:48:53.765515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.765494 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.772320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.772290 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254"] Oct 31 03:48:53.789188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.788823 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b81d65\"/\"openshift-service-ca.crt\"" Oct 31 03:48:53.792852 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.788929 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b81d65\"/\"test-model-b81d65-kserve-self-signed-certs\"" Oct 31 03:48:53.793030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.788994 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b81d65\"/\"kube-root-ca.crt\"" Oct 31 03:48:53.793116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.789096 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b81d65\"/\"default-dockercfg-8frrm\"" Oct 31 03:48:53.838618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.838581 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-tmp-dir\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.838823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.838706 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-dshm\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.838823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.838757 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216f6f37-f90a-45c2-836f-a596db878cf1-tls-certs\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.838964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.838927 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-home\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.838964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.838949 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-model-cache\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.839091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.839018 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmqvn\" (UniqueName: \"kubernetes.io/projected/216f6f37-f90a-45c2-836f-a596db878cf1-kube-api-access-qmqvn\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.942227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942180 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216f6f37-f90a-45c2-836f-a596db878cf1-tls-certs\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.942434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942358 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-home\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.942434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942384 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-model-cache\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.942557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942450 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqvn\" (UniqueName: \"kubernetes.io/projected/216f6f37-f90a-45c2-836f-a596db878cf1-kube-api-access-qmqvn\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.942557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942533 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-tmp-dir\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.942678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942577 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-dshm\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.942990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942962 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-model-cache\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.943098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.942988 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-home\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.943317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.943297 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-tmp-dir\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.945018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.944987 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-dshm\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:53.945818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.945792 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216f6f37-f90a-45c2-836f-a596db878cf1-tls-certs\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:54.000043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:53.999948 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqvn\" (UniqueName: \"kubernetes.io/projected/216f6f37-f90a-45c2-836f-a596db878cf1-kube-api-access-qmqvn\") pod \"test-model-b81d65-kserve-76f84945d7-hg254\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:54.113634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:54.113542 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:54.647185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:54.646298 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254"] Oct 31 03:48:55.231156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.230770 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:48:55.296673 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.296630 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-podinfo\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.297667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.297644 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-socket\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.300683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.300449 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-socket" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:55.300895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.300878 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d3f882-054f-4e82-b27e-1a0c6cfda587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3f882-054f-4e82-b27e-1a0c6cfda587-istiod-ca-cert\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.301786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.301656 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d3f882-054f-4e82-b27e-1a0c6cfda587-istiod-ca-cert" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:55.302166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.301737 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-envoy\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.302361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.302339 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-certs\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.304398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.304293 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-certs" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:55.304630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.304611 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-token\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-token\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.305057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.305039 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-kube-api-access-cp8c7\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-kube-api-access-cp8c7\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.305517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.305493 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-data\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.305809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.305789 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-data" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:55.305998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.305980 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-credential-socket\") pod \"82d3f882-054f-4e82-b27e-1a0c6cfda587\" (UID: \"82d3f882-054f-4e82-b27e-1a0c6cfda587\") " Oct 31 03:48:55.306340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.306311 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-credential-socket" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:55.306867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.306850 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.306952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.306875 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3f882-054f-4e82-b27e-1a0c6cfda587-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.306952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.306888 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.306952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.306901 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.306952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.306915 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.329240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.323652 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-podinfo" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:55.332698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.332666 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-token" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:55.335456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.335431 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-kube-api-access-cp8c7" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "kube-api-access-cp8c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:55.343365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.343341 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-envoy" pod "82d3f882-054f-4e82-b27e-1a0c6cfda587" (UID: "82d3f882-054f-4e82-b27e-1a0c6cfda587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:55.408588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.408556 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.408588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.408586 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.408862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.408597 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8c7\" (UniqueName: \"kubernetes.io/projected/82d3f882-054f-4e82-b27e-1a0c6cfda587-kube-api-access-cp8c7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.408862 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.408608 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3f882-054f-4e82-b27e-1a0c6cfda587-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:55.656806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.656706 5520 generic.go:356] "Generic (PLEG): container finished" podID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerID="78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c" exitCode=0 Oct 31 03:48:55.656806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.656765 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" Oct 31 03:48:55.657061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.656805 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" event={"ID":"82d3f882-054f-4e82-b27e-1a0c6cfda587","Type":"ContainerDied","Data":"78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c"} Oct 31 03:48:55.657061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.656888 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7" event={"ID":"82d3f882-054f-4e82-b27e-1a0c6cfda587","Type":"ContainerDied","Data":"60043e45588fa050d0fc872c0b37cc2ad4132c3b0e3a00de1f9d3826281c5a90"} Oct 31 03:48:55.657061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.656944 5520 scope.go:122] "RemoveContainer" containerID="78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c" Oct 31 03:48:55.660152 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.660116 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" event={"ID":"216f6f37-f90a-45c2-836f-a596db878cf1","Type":"ContainerStarted","Data":"9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f"} Oct 31 03:48:55.660300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.660167 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" event={"ID":"216f6f37-f90a-45c2-836f-a596db878cf1","Type":"ContainerStarted","Data":"996a2d2c40745363a7b0e5f56bf80e55b8b3766a13c46c1f62893ac543b66d15"} Oct 31 03:48:55.680009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.679702 5520 scope.go:122] "RemoveContainer" containerID="4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b" Oct 31 03:48:55.683033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.682981 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" podStartSLOduration=2.682963101 podStartE2EDuration="2.682963101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:55.67757847 +0000 UTC m=+1959.872101608" watchObservedRunningTime="2026-10-31 03:48:55.682963101 +0000 UTC m=+1959.877486224" Oct 31 03:48:55.696747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.696718 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7"] Oct 31 03:48:55.702163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.702139 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-j98v7"] Oct 31 03:48:55.706107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.706089 5520 scope.go:122] "RemoveContainer" containerID="78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c" Oct 31 03:48:55.706537 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:55.706510 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c\": container with ID starting with 78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c not found: ID does not exist" containerID="78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c" Oct 31 03:48:55.706619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.706552 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c"} err="failed to get container status \"78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c\": rpc error: code = NotFound desc = could not find container \"78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c\": container with ID starting with 78f1832beead92eb826860b7e2edae104cd86d2610df1f6bf0e9ba504855116c not found: ID does not exist" Oct 31 03:48:55.706619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.706581 5520 scope.go:122] "RemoveContainer" containerID="4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b" Oct 31 03:48:55.706921 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:48:55.706888 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b\": container with ID starting with 4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b not found: ID does not exist" containerID="4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b" Oct 31 03:48:55.707025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.706928 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b"} err="failed to get container status \"4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b\": rpc error: code = NotFound desc = could not find container \"4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b\": container with ID starting with 4b180de38b7cacad65e06af1c47854a7c670425b5812421605218e8fa4ca097b not found: ID does not exist" Oct 31 03:48:55.924798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:55.924708 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" path="/var/lib/kubelet/pods/82d3f882-054f-4e82-b27e-1a0c6cfda587/volumes" Oct 31 03:48:56.115235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:56.115166 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:48:58.064167 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.064134 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz"] Oct 31 03:48:58.065111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.065086 5520 cpu_manager.go:414] "containerMap: removing container" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" Oct 31 03:48:58.065111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.065111 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" Oct 31 03:48:58.065331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.065130 5520 cpu_manager.go:414] "containerMap: removing container" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" Oct 31 03:48:58.065331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.065137 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" Oct 31 03:48:58.065444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.065378 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" Oct 31 03:48:58.065444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.065397 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d3f882-054f-4e82-b27e-1a0c6cfda587" containerName="istio-proxy" Oct 31 03:48:58.065444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.065419 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="929fc3a7-eaf6-4d34-8754-dec95c4aff02" containerName="istio-proxy" Oct 31 03:48:58.090968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.090737 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.100377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.099072 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ce9b89\"/\"kube-root-ca.crt\"" Oct 31 03:48:58.111183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.111157 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ce9b89\"/\"default-dockercfg-mwxpp\"" Oct 31 03:48:58.111486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.111467 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ce9b89\"/\"openshift-service-ca.crt\"" Oct 31 03:48:58.111727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.111709 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ce9b89\"/\"test-model-ce9b89-kserve-self-signed-certs\"" Oct 31 03:48:58.112642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.112621 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz"] Oct 31 03:48:58.220495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.220464 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-home\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.220722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.220526 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-model-cache\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.220722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.220624 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-tmp-dir\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.220722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.220698 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-dshm\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.220921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.220869 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7m4\" (UniqueName: \"kubernetes.io/projected/61ef4a9f-a824-4884-a338-f5e22314eeee-kube-api-access-jq7m4\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.220921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.220917 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ef4a9f-a824-4884-a338-f5e22314eeee-tls-certs\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.330280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.329559 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7m4\" (UniqueName: \"kubernetes.io/projected/61ef4a9f-a824-4884-a338-f5e22314eeee-kube-api-access-jq7m4\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.330280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.329663 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ef4a9f-a824-4884-a338-f5e22314eeee-tls-certs\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.330280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.329873 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-home\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.330280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.329950 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-model-cache\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.330280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.329989 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-tmp-dir\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.330280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.330040 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-dshm\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.339228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.331037 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-home\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.339228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.332000 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-model-cache\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.339228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.332275 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-tmp-dir\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.389464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.375444 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ef4a9f-a824-4884-a338-f5e22314eeee-tls-certs\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.389464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.375759 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-dshm\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.389464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.378461 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7m4\" (UniqueName: \"kubernetes.io/projected/61ef4a9f-a824-4884-a338-f5e22314eeee-kube-api-access-jq7m4\") pod \"test-model-ce9b89-kserve-8fbfd9b9f-kxpzz\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:58.425762 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:58.425508 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:48:59.028328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:59.028299 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz"] Oct 31 03:48:59.030435 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:48:59.030408 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61ef4a9f_a824_4884_a338_f5e22314eeee.slice/crio-3444a062ddf8c5c15f7b8a3c124549420b21438cedd7d7419cfb1d0b5de25330 WatchSource:0}: Error finding container 3444a062ddf8c5c15f7b8a3c124549420b21438cedd7d7419cfb1d0b5de25330: Status 404 returned error can't find the container with id 3444a062ddf8c5c15f7b8a3c124549420b21438cedd7d7419cfb1d0b5de25330 Oct 31 03:48:59.842193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:59.841988 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" event={"ID":"61ef4a9f-a824-4884-a338-f5e22314eeee","Type":"ContainerStarted","Data":"8565c5e340ed8f276acfa769575b6fd17bd97c350e0c68d3b3c7040ad434580d"} Oct 31 03:48:59.845063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:59.845034 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" event={"ID":"61ef4a9f-a824-4884-a338-f5e22314eeee","Type":"ContainerStarted","Data":"3444a062ddf8c5c15f7b8a3c124549420b21438cedd7d7419cfb1d0b5de25330"} Oct 31 03:48:59.867275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:48:59.867180 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" podStartSLOduration=1.8671446779999998 podStartE2EDuration="1.867144678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:59.862436107 +0000 UTC m=+1964.056959230" watchObservedRunningTime="2026-10-31 03:48:59.867144678 +0000 UTC m=+1964.061667800" Oct 31 03:49:00.445630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:00.436726 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:49:04.125962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:04.124592 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:49:04.174045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:04.174005 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:49:05.098918 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:05.098888 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:49:08.426002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:08.425966 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:49:08.438506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:08.438474 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:49:09.130562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:09.130535 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:49:09.362446 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:49:09.362408 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:49:15.862764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.862721 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554"] Oct 31 03:49:15.879139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.879104 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:15.890215 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.886158 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554"] Oct 31 03:49:15.895242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.890650 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"kube-root-ca.crt\"" Oct 31 03:49:15.895242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.890806 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"default-dockercfg-s9xrm\"" Oct 31 03:49:15.895242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.890864 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"openshift-service-ca.crt\"" Oct 31 03:49:15.895242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.891671 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"auth-test-model-b8fd22-kserve-self-signed-certs\"" Oct 31 03:49:15.913247 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.913185 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2sgn\" (UniqueName: \"kubernetes.io/projected/e86477ac-83fb-4266-a83f-560f10bdb64c-kube-api-access-d2sgn\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:15.913532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.913513 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-dshm\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:15.913669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.913655 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-model-cache\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:15.913968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.913950 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86477ac-83fb-4266-a83f-560f10bdb64c-tls-certs\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:15.914095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.914082 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-home\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:15.914228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:15.914193 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-tmp-dir\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.019051 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-dshm\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.019125 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-model-cache\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.019403 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86477ac-83fb-4266-a83f-560f10bdb64c-tls-certs\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.019448 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-home\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.019481 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-tmp-dir\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.019647 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2sgn\" (UniqueName: \"kubernetes.io/projected/e86477ac-83fb-4266-a83f-560f10bdb64c-kube-api-access-d2sgn\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.021404 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-model-cache\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.022334 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-home\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.024403 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-tmp-dir\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.024776 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"auth-test-model-b8fd22-kserve-self-signed-certs\"" Oct 31 03:49:16.033252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.030917 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"kube-root-ca.crt\"" Oct 31 03:49:16.049494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.049462 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"openshift-service-ca.crt\"" Oct 31 03:49:16.068956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.068882 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86477ac-83fb-4266-a83f-560f10bdb64c-tls-certs\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.069841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.069816 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2sgn\" (UniqueName: \"kubernetes.io/projected/e86477ac-83fb-4266-a83f-560f10bdb64c-kube-api-access-d2sgn\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.150866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.150733 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-dshm\") pod \"auth-test-model-b8fd22-kserve-78f688b749-ph554\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.219751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.219715 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b8fd22\"/\"default-dockercfg-s9xrm\"" Oct 31 03:49:16.227552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.227510 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:16.789890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:16.789805 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554"] Oct 31 03:49:16.794615 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:49:16.794582 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode86477ac_83fb_4266_a83f_560f10bdb64c.slice/crio-5f4fcd6dd98e19a92036dc191a6b5ee84962950b527f94de5f510ecf75150336 WatchSource:0}: Error finding container 5f4fcd6dd98e19a92036dc191a6b5ee84962950b527f94de5f510ecf75150336: Status 404 returned error can't find the container with id 5f4fcd6dd98e19a92036dc191a6b5ee84962950b527f94de5f510ecf75150336 Oct 31 03:49:17.492436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:17.492384 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" event={"ID":"e86477ac-83fb-4266-a83f-560f10bdb64c","Type":"ContainerStarted","Data":"e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c"} Oct 31 03:49:17.492436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:17.492441 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" event={"ID":"e86477ac-83fb-4266-a83f-560f10bdb64c","Type":"ContainerStarted","Data":"5f4fcd6dd98e19a92036dc191a6b5ee84962950b527f94de5f510ecf75150336"} Oct 31 03:49:17.512483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:17.512423 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" podStartSLOduration=2.512403005 podStartE2EDuration="2.512403005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:17.510458592 +0000 UTC m=+1981.704981717" watchObservedRunningTime="2026-10-31 03:49:17.512403005 +0000 UTC m=+1981.706926127" Oct 31 03:49:18.228310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:18.228282 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:26.228452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:26.228359 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:26.242947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:26.242911 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:26.604767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:26.604676 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:49:29.925601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.925565 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf"] Oct 31 03:49:29.932166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.932141 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:29.938920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.938890 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf"] Oct 31 03:49:29.950915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.950888 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-86a834\"/\"auth-test-model-86a834-kserve-self-signed-certs\"" Oct 31 03:49:29.951532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.951502 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-86a834\"/\"openshift-service-ca.crt\"" Oct 31 03:49:29.951708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.951570 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-86a834\"/\"default-dockercfg-grqlb\"" Oct 31 03:49:29.951708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.951638 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-86a834\"/\"kube-root-ca.crt\"" Oct 31 03:49:29.996314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.994771 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-model-cache\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:29.996314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.994832 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tmp-dir\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:29.996314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.994857 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwgw\" (UniqueName: \"kubernetes.io/projected/c13fcad4-2c72-4249-b39c-5ae3d819fd20-kube-api-access-rqwgw\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:29.996314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.995083 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-home\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:29.996314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.995181 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tls-certs\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:29.996314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:29.995312 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-dshm\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.096624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.096576 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-model-cache\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.096624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.096614 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tmp-dir\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.096907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.096647 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwgw\" (UniqueName: \"kubernetes.io/projected/c13fcad4-2c72-4249-b39c-5ae3d819fd20-kube-api-access-rqwgw\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.096907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.096897 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-home\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.097024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.096990 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tls-certs\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.097024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.097011 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-model-cache\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.097143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.097117 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-dshm\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.098032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.098004 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tmp-dir\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.098581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.098560 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-home\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.100373 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.100348 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-dshm\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.102242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.102220 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tls-certs\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.106265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.106197 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwgw\" (UniqueName: \"kubernetes.io/projected/c13fcad4-2c72-4249-b39c-5ae3d819fd20-kube-api-access-rqwgw\") pod \"auth-test-model-86a834-kserve-58d79bdbb9-ql6wf\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.276228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.275597 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:30.875764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:30.875725 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf"] Oct 31 03:49:31.684559 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:31.684518 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" event={"ID":"c13fcad4-2c72-4249-b39c-5ae3d819fd20","Type":"ContainerStarted","Data":"2df23654114c354e0e7a28279dd2a9d20f7eb54268bf5c12163c0f7f7bf64486"} Oct 31 03:49:31.685079 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:31.684576 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" event={"ID":"c13fcad4-2c72-4249-b39c-5ae3d819fd20","Type":"ContainerStarted","Data":"4458cf654ed24332223137d96736a5c731c3e1cee81110329e5b9bd8163c6e41"} Oct 31 03:49:31.701646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:31.701568 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" podStartSLOduration=2.7015508329999998 podStartE2EDuration="2.701550833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:31.69991818 +0000 UTC m=+1995.894441303" watchObservedRunningTime="2026-10-31 03:49:31.701550833 +0000 UTC m=+1995.896073957" Oct 31 03:49:32.276460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:32.276425 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:34.133226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.126946 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s"] Oct 31 03:49:34.149277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.145264 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s"] Oct 31 03:49:34.149277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.145368 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.149763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.149741 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-63258b\"/\"default-dockercfg-b7px6\"" Oct 31 03:49:34.149924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.149909 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-63258b\"/\"kube-root-ca.crt\"" Oct 31 03:49:34.150003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.149971 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-63258b\"/\"openshift-service-ca.crt\"" Oct 31 03:49:34.150068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.150031 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-63258b\"/\"auth-test-model-63258b-kserve-self-signed-certs\"" Oct 31 03:49:34.221960 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.221919 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-dshm\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.222156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.222113 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tmp-dir\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.222156 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.222149 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4r6c\" (UniqueName: \"kubernetes.io/projected/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-kube-api-access-b4r6c\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.222258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.222177 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-model-cache\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.222368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.222351 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tls-certs\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.222615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.222598 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-home\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.345231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.344569 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-dshm\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.345231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.344747 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tmp-dir\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.345231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.344789 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4r6c\" (UniqueName: \"kubernetes.io/projected/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-kube-api-access-b4r6c\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.345231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.344828 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-model-cache\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.345231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.344976 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tls-certs\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.345231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.345108 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-home\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.356237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.345846 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-home\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.356237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.346849 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tmp-dir\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.356237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.347084 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-model-cache\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.365279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.364443 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tls-certs\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.380631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.380595 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4r6c\" (UniqueName: \"kubernetes.io/projected/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-kube-api-access-b4r6c\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.386631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.386555 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-dshm\") pod \"auth-test-model-63258b-kserve-96fb788fc-c7x6s\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.469238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.466726 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:34.930275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:34.930166 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s"] Oct 31 03:49:34.930649 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:49:34.930620 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd76db6e7_f346_42cc_a8bf_c423ab6ab4f6.slice/crio-15f84b4866649b5c17ab215c5f8f8f0b749dc6172a48528d3b74b6b61f2122e8 WatchSource:0}: Error finding container 15f84b4866649b5c17ab215c5f8f8f0b749dc6172a48528d3b74b6b61f2122e8: Status 404 returned error can't find the container with id 15f84b4866649b5c17ab215c5f8f8f0b749dc6172a48528d3b74b6b61f2122e8 Oct 31 03:49:35.759907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:35.759854 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" event={"ID":"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6","Type":"ContainerStarted","Data":"3cded62a839653c584eeef05ed4462c3ec15fa3337f8ef9c6529cecda6af37ec"} Oct 31 03:49:35.760535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:35.759957 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" event={"ID":"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6","Type":"ContainerStarted","Data":"15f84b4866649b5c17ab215c5f8f8f0b749dc6172a48528d3b74b6b61f2122e8"} Oct 31 03:49:35.785265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:35.783143 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" podStartSLOduration=1.783128254 podStartE2EDuration="1.783128254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:35.780440279 +0000 UTC m=+1999.974963413" watchObservedRunningTime="2026-10-31 03:49:35.783128254 +0000 UTC m=+1999.977651392" Oct 31 03:49:36.472231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:36.468339 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:39.362187 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:49:39.362140 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:49:40.276807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:40.276774 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:40.291880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:40.291812 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:40.829740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:40.829710 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:49:44.470089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:44.470051 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:44.510063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:44.510036 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:44.932724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:44.932697 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:49:52.018225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.006382 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:52.067143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.066154 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:52.067143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.066863 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": context deadline exceeded" Oct 31 03:49:52.072303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.072267 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:52.152993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.152850 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:52.222345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.222299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 03:49:52.335280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.334666 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": context deadline exceeded" Oct 31 03:49:52.335280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.334988 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": context deadline exceeded" Oct 31 03:49:52.335280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.335239 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": context deadline exceeded" Oct 31 03:49:52.352720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.352302 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:52.380428 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.380379 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:52.481227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.478940 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 03:49:52.486684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.486633 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": context deadline exceeded" Oct 31 03:49:52.540274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.505427 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 03:49:52.561815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.561772 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:52.989242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:52.986496 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" podUID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.217:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.016226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.013299 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.016226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.013872 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:49:53.077770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.077718 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.128624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.080829 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.129248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.129222 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:49:53.129453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.129441 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:49:53.161343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.161298 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.214:8000/ready\": context deadline exceeded" Oct 31 03:49:53.183223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.080891 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": context deadline exceeded" Oct 31 03:49:53.229841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.195741 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:49:53.230151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.230126 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:49:53.232569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.222781 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.233887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.233786 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:49:53.335909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.335797 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": context deadline exceeded" Oct 31 03:49:53.336274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.336242 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:49:53.455227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.383507 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Oct 31 03:49:53.494229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.402435 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 31 03:49:53.502070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.403871 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.503174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.416308 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.182:8000/ready\": context deadline exceeded" Oct 31 03:49:53.503665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.469767 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.504060 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.496295 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:49:53.504343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.504325 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:49:53.504564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.504551 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:49:53.558459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.551411 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.602974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.602870 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" podUID="e86477ac-83fb-4266-a83f-560f10bdb64c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.682143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.682111 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:49:53.972842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:53.882294 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 31 03:49:54.034711 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.034681 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:49:54.045225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.044597 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 03:49:54.046830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.046801 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:54.066249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.066219 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:49:54.066603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.066589 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:49:54.193652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.193613 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:54.296291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.292752 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:49:54.296291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.292820 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:49:54.296291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.292854 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:49:54.296291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.292884 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:49:54.480278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.460900 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:54.750269 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:49:54.750237 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:50:05.022457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:05.022372 5520 generic.go:356] "Generic (PLEG): container finished" podID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerID="6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7" exitCode=0 Oct 31 03:50:05.022457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:05.022442 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" event={"ID":"f2aec880-cd9a-46fb-99a6-4a51e8921c94","Type":"ContainerDied","Data":"6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7"} Oct 31 03:50:05.023018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:05.022919 5520 scope.go:122] "RemoveContainer" containerID="6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7" Oct 31 03:50:05.024812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:05.024784 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:05.024904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:05.024851 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:05.024904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:05.024887 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:05.474261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:05.474232 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:50:06.031978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.031944 5520 generic.go:356] "Generic (PLEG): container finished" podID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerID="75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8" exitCode=0 Oct 31 03:50:06.032482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.032013 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" event={"ID":"fcb74b35-d0cc-4865-9617-263b41b4ec7a","Type":"ContainerDied","Data":"75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8"} Oct 31 03:50:06.032642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.032622 5520 scope.go:122] "RemoveContainer" containerID="75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8" Oct 31 03:50:06.034579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.034553 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" event={"ID":"f2aec880-cd9a-46fb-99a6-4a51e8921c94","Type":"ContainerStarted","Data":"cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821"} Oct 31 03:50:06.043822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.043780 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:06.043985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.043962 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:06.044114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.044089 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:06.791226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.776267 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:50:06.791226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.776320 5520 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 31 03:50:06.791226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:06.776389 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 31 03:50:07.047884 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:07.047806 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" event={"ID":"fcb74b35-d0cc-4865-9617-263b41b4ec7a","Type":"ContainerStarted","Data":"9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc"} Oct 31 03:50:07.050558 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:07.050506 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:50:07.051742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:07.051707 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 03:50:07.051831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:07.051770 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 03:50:07.789229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:07.787323 5520 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 31 03:50:07.789229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:07.787404 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 31 03:50:08.049287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:08.048659 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 03:50:08.049287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:08.048717 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 03:50:08.782226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:08.775956 5520 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 31 03:50:08.782226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:08.776021 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 31 03:50:09.049531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:09.049439 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 03:50:09.049531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:09.049504 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 03:50:09.365130 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:50:09.365037 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:50:09.788817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:09.788782 5520 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 31 03:50:09.789099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:09.789073 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 31 03:50:10.048990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:10.048907 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 03:50:10.048990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:10.048970 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 03:50:10.083238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:10.082309 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:50:10.786230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:10.785652 5520 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 31 03:50:10.786230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:10.785725 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 31 03:50:11.049708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:11.049615 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 03:50:11.049965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:11.049942 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 03:50:11.787226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:11.786176 5520 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 31 03:50:11.787226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:11.786285 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 31 03:50:12.049507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:12.049410 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 03:50:12.049789 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:12.049758 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 03:50:12.777324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:12.777290 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:50:12.777865 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:12.777839 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:50:12.778609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:12.778591 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:50:13.059838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:13.053921 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 03:50:13.059838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:13.053992 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 03:50:14.054317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:14.054286 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:50:14.055530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:14.055507 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:50:31.561932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:31.561881 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs"] Oct 31 03:50:37.760440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.760399 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr"] Oct 31 03:50:37.764576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.764547 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.779493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.779461 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr"] Oct 31 03:50:37.824952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.824915 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5883cc3-a2f9-4082-bd89-f0940ae80531-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825077 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825220 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825259 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825384 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825477 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825594 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825576 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825681 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtljg\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-kube-api-access-dtljg\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.825820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.825801 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.859166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.859129 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-afb8be-openshift-default-dockercfg-gvlck\"" Oct 31 03:50:37.940730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.940684 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.940943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.940820 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.940943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.940859 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.940943 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.940930 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.941116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.940996 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.941116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.941096 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.941283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.941194 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtljg\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-kube-api-access-dtljg\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.941352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.941301 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.942262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.942099 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5883cc3-a2f9-4082-bd89-f0940ae80531-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.942262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.941716 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.942262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.941767 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.942262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.941819 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.942262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.941522 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.942861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.942835 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5883cc3-a2f9-4082-bd89-f0940ae80531-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.949787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.949480 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.950094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.950073 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.950620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.950596 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:37.950698 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:37.950661 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtljg\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-kube-api-access-dtljg\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:38.110211 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.110112 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:38.405482 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:50:38.405449 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5883cc3_a2f9_4082_bd89_f0940ae80531.slice/crio-0ba13b4045b2d3fee186d1b3f4018f29a006aeddb3b939ea45f3398d75920f7e WatchSource:0}: Error finding container 0ba13b4045b2d3fee186d1b3f4018f29a006aeddb3b939ea45f3398d75920f7e: Status 404 returned error can't find the container with id 0ba13b4045b2d3fee186d1b3f4018f29a006aeddb3b939ea45f3398d75920f7e Oct 31 03:50:38.406283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.406237 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr"] Oct 31 03:50:38.409353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.409317 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:38.409475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.409397 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:38.409475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.409442 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:38.924241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.922549 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" event={"ID":"e5883cc3-a2f9-4082-bd89-f0940ae80531","Type":"ContainerStarted","Data":"6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a"} Oct 31 03:50:38.924241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.922619 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" event={"ID":"e5883cc3-a2f9-4082-bd89-f0940ae80531","Type":"ContainerStarted","Data":"0ba13b4045b2d3fee186d1b3f4018f29a006aeddb3b939ea45f3398d75920f7e"} Oct 31 03:50:38.946219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:38.946140 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podStartSLOduration=1.9461220799999999 podStartE2EDuration="1.94612208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:38.94375305 +0000 UTC m=+2063.138276190" watchObservedRunningTime="2026-10-31 03:50:38.94612208 +0000 UTC m=+2063.140645202" Oct 31 03:50:39.110798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:39.110759 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:39.112774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:39.112748 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 31 03:50:39.112901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:39.112806 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 31 03:50:39.362975 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:50:39.362938 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:50:40.110984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:40.110944 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 31 03:50:40.111437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:40.111023 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 31 03:50:41.115192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:41.111854 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 31 03:50:41.115192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:41.111915 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 31 03:50:42.111196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:42.111155 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 31 03:50:42.111538 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:42.111260 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 31 03:50:43.112666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:43.112628 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 31 03:50:43.113357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:43.113318 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 31 03:50:44.111592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:44.111553 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 31 03:50:44.111902 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:44.111871 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 31 03:50:44.513853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:44.513818 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452"] Oct 31 03:50:45.110802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:45.110763 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 31 03:50:45.111010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:45.110841 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 31 03:50:46.116763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:46.116724 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:46.117353 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:46.117323 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:46.118317 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:46.118299 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:50:46.623408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:46.623364 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" containerID="cri-o://a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f" gracePeriod=45 Oct 31 03:50:46.950396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:46.950367 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:50:47.037065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.037024 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-home\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-home\") pod \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " Oct 31 03:50:47.037312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.037291 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-kube-api-access-vl59d\" (UniqueName: \"kubernetes.io/projected/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-kube-api-access-vl59d\") pod \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " Oct 31 03:50:47.037393 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.037350 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-home" pod "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" (UID: "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.037699 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tmp-dir\") pod \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.037800 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-model-cache\") pod \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.037874 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tls-certs\") pod \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.037907 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-dshm\") pod \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\" (UID: \"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c\") " Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.038504 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-model-cache" pod "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" (UID: "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.038895 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.038914 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:47.039446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.039135 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tmp-dir" pod "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" (UID: "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:47.042337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.042117 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-dshm" pod "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" (UID: "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:47.042337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.042290 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tls-certs" pod "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" (UID: "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:50:47.042695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.042671 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-kube-api-access-vl59d" pod "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" (UID: "1e5ffadc-df64-44f8-a33c-bb9c2960ca0c"). InnerVolumeSpecName "kube-api-access-vl59d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:47.049067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.049035 5520 generic.go:356] "Generic (PLEG): container finished" podID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerID="a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f" exitCode=0 Oct 31 03:50:47.056348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.050280 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" event={"ID":"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c","Type":"ContainerDied","Data":"a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f"} Oct 31 03:50:47.056348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.050321 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" Oct 31 03:50:47.056348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.050338 5520 scope.go:122] "RemoveContainer" containerID="a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f" Oct 31 03:50:47.056348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.050325 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs" event={"ID":"1e5ffadc-df64-44f8-a33c-bb9c2960ca0c","Type":"ContainerDied","Data":"00d135756c1a0b2ba53bf8bb99398a56413051979bd9ada5dbbed6cb5fc84eaa"} Oct 31 03:50:47.093070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.093048 5520 scope.go:122] "RemoveContainer" containerID="a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f" Oct 31 03:50:47.093477 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:50:47.093452 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f\": container with ID starting with a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f not found: ID does not exist" containerID="a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f" Oct 31 03:50:47.093579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.093489 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f"} err="failed to get container status \"a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f\": rpc error: code = NotFound desc = could not find container \"a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f\": container with ID starting with a7a78d7336c95d537682db9a21d5106fb86785f6677942415954afb85441df2f not found: ID does not exist" Oct 31 03:50:47.102375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.102349 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs"] Oct 31 03:50:47.111738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.111713 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-034fe8/test-model-034fe8-kserve-67cfc86bf9-rrsrs"] Oct 31 03:50:47.141173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.141145 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:47.141173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.141170 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:47.141624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.141180 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:47.141624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.141190 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl59d\" (UniqueName: \"kubernetes.io/projected/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c-kube-api-access-vl59d\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:47.920480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:47.920449 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" path="/var/lib/kubelet/pods/1e5ffadc-df64-44f8-a33c-bb9c2960ca0c/volumes" Oct 31 03:50:52.743938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.743898 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq"] Oct 31 03:50:52.744389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.744132 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" containerID="cri-o://f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9" gracePeriod=30 Oct 31 03:50:52.749171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.749140 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb"] Oct 31 03:50:52.749425 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.749404 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" containerID="cri-o://8781a06db00ee77c66d14a701d775f0d0991383bb2a8bff3d560e6c670206406" gracePeriod=30 Oct 31 03:50:52.858537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.858503 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs"] Oct 31 03:50:52.860036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.860009 5520 cpu_manager.go:414] "containerMap: removing container" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" Oct 31 03:50:52.860293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.860195 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" Oct 31 03:50:52.860867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.860851 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5ffadc-df64-44f8-a33c-bb9c2960ca0c" containerName="main" Oct 31 03:50:52.873419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.873369 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.888905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.888758 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs"] Oct 31 03:50:52.954132 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954084 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954286 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwm8\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-kube-api-access-rqwm8\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954435 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954404 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954493 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954554 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954697 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954677 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be417875-9d21-454c-908f-d5b154479635-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954780 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954912 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954896 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be417875-9d21-454c-908f-d5b154479635-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:52.954989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:52.954969 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.056524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056430 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwm8\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-kube-api-access-rqwm8\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.056705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056538 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.056705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056601 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.056705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056651 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.056900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056718 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be417875-9d21-454c-908f-d5b154479635-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.056900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056791 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056922 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be417875-9d21-454c-908f-d5b154479635-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.057001 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.057046 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.057087 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.057180 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.056998 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.057347 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.057668 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.057645 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be417875-9d21-454c-908f-d5b154479635-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.059169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.059145 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.059369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.059350 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be417875-9d21-454c-908f-d5b154479635-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.065622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.065600 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.065766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.065743 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwm8\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-kube-api-access-rqwm8\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.215603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.215563 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:53.507640 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:50:53.507607 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe417875_9d21_454c_908f_d5b154479635.slice/crio-0269fb5874195eb75d93bec50dc3f1521547b278a1daa249462d8e2381f5ee02 WatchSource:0}: Error finding container 0269fb5874195eb75d93bec50dc3f1521547b278a1daa249462d8e2381f5ee02: Status 404 returned error can't find the container with id 0269fb5874195eb75d93bec50dc3f1521547b278a1daa249462d8e2381f5ee02 Oct 31 03:50:53.509668 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.509634 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs"] Oct 31 03:50:53.511747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.511709 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:53.511858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.511791 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:53.511858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:53.511830 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:50:54.125182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:54.125141 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" event={"ID":"be417875-9d21-454c-908f-d5b154479635","Type":"ContainerStarted","Data":"e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c"} Oct 31 03:50:54.125182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:54.125215 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" event={"ID":"be417875-9d21-454c-908f-d5b154479635","Type":"ContainerStarted","Data":"0269fb5874195eb75d93bec50dc3f1521547b278a1daa249462d8e2381f5ee02"} Oct 31 03:50:54.151333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:54.151266 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podStartSLOduration=2.151246931 podStartE2EDuration="2.151246931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:54.143636749 +0000 UTC m=+2078.338159887" watchObservedRunningTime="2026-10-31 03:50:54.151246931 +0000 UTC m=+2078.345770054" Oct 31 03:50:54.216601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:54.216564 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:50:54.218848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:54.218821 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 03:50:54.218992 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:54.218874 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 03:50:55.216339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:55.216298 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 03:50:55.216780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:55.216355 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 03:50:56.215908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:56.215867 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 03:50:56.216095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:56.215929 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 03:50:57.216099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:57.216061 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 03:50:57.216533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:57.216131 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 03:50:58.199252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.198910 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:50:58.214086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.212529 5520 generic.go:356] "Generic (PLEG): container finished" podID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerID="8781a06db00ee77c66d14a701d775f0d0991383bb2a8bff3d560e6c670206406" exitCode=0 Oct 31 03:50:58.214086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.213994 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" event={"ID":"004d9602-8bbe-49f3-a808-97c335b0b1d8","Type":"ContainerDied","Data":"8781a06db00ee77c66d14a701d775f0d0991383bb2a8bff3d560e6c670206406"} Oct 31 03:50:58.216786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.216718 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 03:50:58.216786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.216773 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 03:50:58.218448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.217856 5520 generic.go:356] "Generic (PLEG): container finished" podID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerID="f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9" exitCode=0 Oct 31 03:50:58.218448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.217998 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" event={"ID":"de000444-58e9-4de9-b340-4b8ec8b9d583","Type":"ContainerDied","Data":"f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9"} Oct 31 03:50:58.218448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.218048 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" event={"ID":"de000444-58e9-4de9-b340-4b8ec8b9d583","Type":"ContainerDied","Data":"0918313846dad315a8e9cb54545b264bacab1664b968c3130a7108e88e3f48a6"} Oct 31 03:50:58.218448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.218078 5520 scope.go:122] "RemoveContainer" containerID="f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9" Oct 31 03:50:58.218448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.218287 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq" Oct 31 03:50:58.234402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.234384 5520 scope.go:122] "RemoveContainer" containerID="f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9" Oct 31 03:50:58.234815 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:50:58.234797 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9\": container with ID starting with f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9 not found: ID does not exist" containerID="f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9" Oct 31 03:50:58.235032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.234980 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9"} err="failed to get container status \"f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9\": rpc error: code = NotFound desc = could not find container \"f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9\": container with ID starting with f498f2e579500071b90cbd425dca85cde8c454c0468ff623816451067aaa70f9 not found: ID does not exist" Oct 31 03:50:58.305026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.304987 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-envoy\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305056 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de000444-58e9-4de9-b340-4b8ec8b9d583-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de000444-58e9-4de9-b340-4b8ec8b9d583-istiod-ca-cert\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305122 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-token\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-token\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305185 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-data\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305272 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-podinfo\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305509 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-credential-socket\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305581 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-socket\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305690 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-certs\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.305819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.305729 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-kube-api-access-w5z7z\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-kube-api-access-w5z7z\") pod \"de000444-58e9-4de9-b340-4b8ec8b9d583\" (UID: \"de000444-58e9-4de9-b340-4b8ec8b9d583\") " Oct 31 03:50:58.306842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.306814 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-credential-socket" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.307267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.307242 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-data" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.308360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.308333 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-socket" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.308526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.308505 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-certs" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.311281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311189 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-kube-api-access-w5z7z" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "kube-api-access-w5z7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311425 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de000444-58e9-4de9-b340-4b8ec8b9d583-istiod-ca-cert" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311666 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311691 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311705 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311719 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311732 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5z7z\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-kube-api-access-w5z7z\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.311747 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de000444-58e9-4de9-b340-4b8ec8b9d583-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.315467 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-envoy" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.318740 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.315544 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-podinfo" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:58.319445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.318775 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-token" pod "de000444-58e9-4de9-b340-4b8ec8b9d583" (UID: "de000444-58e9-4de9-b340-4b8ec8b9d583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:58.339915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.339534 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:50:58.415467 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415376 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-podinfo\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.415658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415604 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-kube-api-access-nvlbz\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-kube-api-access-nvlbz\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.415735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415704 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-token\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-token\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.415806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415756 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-socket\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.415806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415787 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-credential-socket\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.415927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415869 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-data\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.415927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415903 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-certs\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.416045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415936 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004d9602-8bbe-49f3-a808-97c335b0b1d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d9602-8bbe-49f3-a808-97c335b0b1d8-istiod-ca-cert\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.416045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.415964 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-envoy\") pod \"004d9602-8bbe-49f3-a808-97c335b0b1d8\" (UID: \"004d9602-8bbe-49f3-a808-97c335b0b1d8\") " Oct 31 03:50:58.417027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.417001 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.417182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.417031 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.417182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.417047 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de000444-58e9-4de9-b340-4b8ec8b9d583-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.417635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.417605 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-socket" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.418462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.418423 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-credential-socket" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.418462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.418433 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-certs" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.418704 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.418679 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004d9602-8bbe-49f3-a808-97c335b0b1d8-istiod-ca-cert" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:50:58.418793 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.418190 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-data" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.421605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.421575 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-kube-api-access-nvlbz" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "kube-api-access-nvlbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:58.421988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.421951 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-podinfo" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:50:58.424018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.423982 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-token" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:50:58.426536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.426462 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-envoy" pod "004d9602-8bbe-49f3-a808-97c335b0b1d8" (UID: "004d9602-8bbe-49f3-a808-97c335b0b1d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:50:58.518734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518690 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.518734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518731 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.518734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518742 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.519075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518752 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.519075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518760 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.519075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518771 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004d9602-8bbe-49f3-a808-97c335b0b1d8-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.519075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518784 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.519075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518792 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004d9602-8bbe-49f3-a808-97c335b0b1d8-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.519075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.518800 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvlbz\" (UniqueName: \"kubernetes.io/projected/004d9602-8bbe-49f3-a808-97c335b0b1d8-kube-api-access-nvlbz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:50:58.547691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.547616 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq"] Oct 31 03:50:58.558834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:58.553669 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-gk7nq"] Oct 31 03:50:59.216418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.216390 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 31 03:50:59.216602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.216447 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 31 03:50:59.228760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.228732 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" Oct 31 03:50:59.228760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.228753 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb" event={"ID":"004d9602-8bbe-49f3-a808-97c335b0b1d8","Type":"ContainerDied","Data":"cff2349dd93a24aa19881d5c509e7ccaddc20e592adfaa36600be0b90371c2ef"} Oct 31 03:50:59.229259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.228812 5520 scope.go:122] "RemoveContainer" containerID="8781a06db00ee77c66d14a701d775f0d0991383bb2a8bff3d560e6c670206406" Oct 31 03:50:59.289229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.288258 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb"] Oct 31 03:50:59.304417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.304387 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kc9wb"] Oct 31 03:50:59.587431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.581751 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" containerID="cri-o://9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9" gracePeriod=45 Oct 31 03:50:59.920718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.920686 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:50:59.922537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.922511 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" path="/var/lib/kubelet/pods/004d9602-8bbe-49f3-a808-97c335b0b1d8/volumes" Oct 31 03:50:59.923188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:50:59.923165 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" path="/var/lib/kubelet/pods/de000444-58e9-4de9-b340-4b8ec8b9d583/volumes" Oct 31 03:51:00.048274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.047318 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-dshm\") pod \"d47568df-5e8e-477e-a372-dd8710ab78d9\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " Oct 31 03:51:00.048274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.047460 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-model-cache\") pod \"d47568df-5e8e-477e-a372-dd8710ab78d9\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " Oct 31 03:51:00.048274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.047504 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47568df-5e8e-477e-a372-dd8710ab78d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d47568df-5e8e-477e-a372-dd8710ab78d9-tls-certs\") pod \"d47568df-5e8e-477e-a372-dd8710ab78d9\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " Oct 31 03:51:00.048274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.047545 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47568df-5e8e-477e-a372-dd8710ab78d9-kube-api-access-6cmn2\" (UniqueName: \"kubernetes.io/projected/d47568df-5e8e-477e-a372-dd8710ab78d9-kube-api-access-6cmn2\") pod \"d47568df-5e8e-477e-a372-dd8710ab78d9\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " Oct 31 03:51:00.048274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.047822 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-model-cache" pod "d47568df-5e8e-477e-a372-dd8710ab78d9" (UID: "d47568df-5e8e-477e-a372-dd8710ab78d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:00.048274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.047884 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-home\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-home\") pod \"d47568df-5e8e-477e-a372-dd8710ab78d9\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " Oct 31 03:51:00.048274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.047980 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-tmp-dir\") pod \"d47568df-5e8e-477e-a372-dd8710ab78d9\" (UID: \"d47568df-5e8e-477e-a372-dd8710ab78d9\") " Oct 31 03:51:00.050345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.049510 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:00.050345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.050310 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-tmp-dir" pod "d47568df-5e8e-477e-a372-dd8710ab78d9" (UID: "d47568df-5e8e-477e-a372-dd8710ab78d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:00.050554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.050492 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-dshm" pod "d47568df-5e8e-477e-a372-dd8710ab78d9" (UID: "d47568df-5e8e-477e-a372-dd8710ab78d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:00.050818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.050798 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-home" pod "d47568df-5e8e-477e-a372-dd8710ab78d9" (UID: "d47568df-5e8e-477e-a372-dd8710ab78d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:51:00.054680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.054651 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47568df-5e8e-477e-a372-dd8710ab78d9-tls-certs" pod "d47568df-5e8e-477e-a372-dd8710ab78d9" (UID: "d47568df-5e8e-477e-a372-dd8710ab78d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:51:00.058347 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.058316 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47568df-5e8e-477e-a372-dd8710ab78d9-kube-api-access-6cmn2" pod "d47568df-5e8e-477e-a372-dd8710ab78d9" (UID: "d47568df-5e8e-477e-a372-dd8710ab78d9"). InnerVolumeSpecName "kube-api-access-6cmn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:51:00.151057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.150964 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:00.151057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.150994 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47568df-5e8e-477e-a372-dd8710ab78d9-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:00.151057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.151009 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmn2\" (UniqueName: \"kubernetes.io/projected/d47568df-5e8e-477e-a372-dd8710ab78d9-kube-api-access-6cmn2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:00.151057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.151020 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:00.151057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.151030 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47568df-5e8e-477e-a372-dd8710ab78d9-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:51:00.221080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.221049 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:51:00.221489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.221465 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:51:00.222318 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.222300 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:51:00.240000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.239970 5520 generic.go:356] "Generic (PLEG): container finished" podID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerID="9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9" exitCode=0 Oct 31 03:51:00.240607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.240044 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" Oct 31 03:51:00.240607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.240134 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" event={"ID":"d47568df-5e8e-477e-a372-dd8710ab78d9","Type":"ContainerDied","Data":"9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9"} Oct 31 03:51:00.240607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.240192 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452" event={"ID":"d47568df-5e8e-477e-a372-dd8710ab78d9","Type":"ContainerDied","Data":"5b0d20af8e99c7d8e59f230f29458c66c61501598552cc50a14ba35ac1af3d1f"} Oct 31 03:51:00.240607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.240260 5520 scope.go:122] "RemoveContainer" containerID="9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9" Oct 31 03:51:00.253826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.253805 5520 scope.go:122] "RemoveContainer" containerID="9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9" Oct 31 03:51:00.254116 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:51:00.254096 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9\": container with ID starting with 9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9 not found: ID does not exist" containerID="9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9" Oct 31 03:51:00.254174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.254128 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9"} err="failed to get container status \"9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9\": rpc error: code = NotFound desc = could not find container \"9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9\": container with ID starting with 9d7cad70e704a68747b306b578c50335890e17826f6e7c91964a832c631d1ec9 not found: ID does not exist" Oct 31 03:51:00.286749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.286715 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452"] Oct 31 03:51:00.293948 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:00.293916 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-46ebfe/test-model-46ebfe-kserve-68466fbfbc-7d452"] Oct 31 03:51:01.921750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:01.921720 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" path="/var/lib/kubelet/pods/d47568df-5e8e-477e-a372-dd8710ab78d9/volumes" Oct 31 03:51:04.024940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:04.024902 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5"] Oct 31 03:51:04.029189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:04.029160 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a2cd1fc-nccg5"] Oct 31 03:51:05.922065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:05.922022 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2385df23-2dd4-429f-ae01-47a8cf70977a" path="/var/lib/kubelet/pods/2385df23-2dd4-429f-ae01-47a8cf70977a/volumes" Oct 31 03:51:07.796640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.796604 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2"] Oct 31 03:51:07.797464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797449 5520 cpu_manager.go:414] "containerMap: removing container" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" Oct 31 03:51:07.797534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797465 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" Oct 31 03:51:07.797534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797480 5520 cpu_manager.go:414] "containerMap: removing container" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" Oct 31 03:51:07.797534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797485 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" Oct 31 03:51:07.797534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797520 5520 cpu_manager.go:414] "containerMap: removing container" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" Oct 31 03:51:07.797534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797525 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" Oct 31 03:51:07.797692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797651 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="004d9602-8bbe-49f3-a808-97c335b0b1d8" containerName="istio-proxy" Oct 31 03:51:07.797692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797667 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47568df-5e8e-477e-a372-dd8710ab78d9" containerName="main" Oct 31 03:51:07.797692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.797674 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="de000444-58e9-4de9-b340-4b8ec8b9d583" containerName="istio-proxy" Oct 31 03:51:07.801543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.801519 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.815878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.815848 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2"] Oct 31 03:51:07.888120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888073 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888240 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzqs\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-kube-api-access-8rzqs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888295 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888432 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888482 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888629 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888721 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888863 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888847 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.888910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.888887 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.990183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990144 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.990398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990248 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.990398 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990294 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.990531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990405 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.990531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990435 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.990687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990661 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.990914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990894 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzqs\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-kube-api-access-8rzqs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.991278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.991260 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.991458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.991430 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.991579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.991287 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.991579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.990974 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.991579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.991554 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.991891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.991867 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.992191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.992169 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.992985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.992961 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.993362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.993340 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.999062 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.999035 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzqs\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-kube-api-access-8rzqs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:07.999479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:07.999459 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:08.128465 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:08.128374 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:08.430934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:08.430907 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2"] Oct 31 03:51:08.431995 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:51:08.431968 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c92f1df_97ea_4f0f_8b10_3c04f109f33c.slice/crio-5e4ca64fd47881d3bac71900dbfac6b4a3b42c13643a3b015950f0baae38df89 WatchSource:0}: Error finding container 5e4ca64fd47881d3bac71900dbfac6b4a3b42c13643a3b015950f0baae38df89: Status 404 returned error can't find the container with id 5e4ca64fd47881d3bac71900dbfac6b4a3b42c13643a3b015950f0baae38df89 Oct 31 03:51:08.435616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:08.435584 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:51:08.435712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:08.435647 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:51:08.435712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:08.435679 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:51:09.331484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:09.331443 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" event={"ID":"8c92f1df-97ea-4f0f-8b10-3c04f109f33c","Type":"ContainerStarted","Data":"1ff892b288093ad0817ce15551c3e5de79c51d8380a54883765a4d85252e3c53"} Oct 31 03:51:09.332025 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:09.331508 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" event={"ID":"8c92f1df-97ea-4f0f-8b10-3c04f109f33c","Type":"ContainerStarted","Data":"5e4ca64fd47881d3bac71900dbfac6b4a3b42c13643a3b015950f0baae38df89"} Oct 31 03:51:09.356471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:09.356407 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podStartSLOduration=2.356387345 podStartE2EDuration="2.356387345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:09.350261677 +0000 UTC m=+2093.544784798" watchObservedRunningTime="2026-10-31 03:51:09.356387345 +0000 UTC m=+2093.550910467" Oct 31 03:51:09.363584 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:51:09.363551 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:51:10.128734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:10.128700 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:10.130212 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:10.130177 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:10.130336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:10.130241 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:11.129720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:11.129683 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:11.130154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:11.129761 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:12.129154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:12.129117 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:12.129363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:12.129196 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:13.129652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:13.129619 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:13.130196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:13.129682 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:14.129189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:14.129159 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:14.129414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:14.129260 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:15.128942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:15.128902 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:15.129408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:15.128971 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:16.129311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:16.129279 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:16.129807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:16.129337 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:17.129386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:17.129352 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:17.129879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:17.129413 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:18.129297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:18.129259 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:18.129487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:18.129328 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:19.128774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:19.128737 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:19.128959 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:19.128790 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:20.128976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:20.128936 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:20.129409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:20.129002 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:21.129401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:21.129363 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:21.129810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:21.129416 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:22.129630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:22.129588 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:22.130158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:22.129669 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:23.128662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:23.128618 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:23.128893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:23.128851 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:23.128978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:23.128907 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:24.135616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:24.135580 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:24.136272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:24.136238 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:25.128798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:25.128759 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:25.128996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:25.128827 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:26.128912 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:26.128881 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:26.129473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:26.128951 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:27.129518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:27.129479 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:27.130109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:27.129537 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:28.129529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:28.129490 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:28.130137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:28.129564 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:29.129661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:29.129622 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:29.130091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:29.129703 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:30.128723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:30.128685 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:30.128946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:30.128752 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:31.129426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:31.129390 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:31.129971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:31.129450 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:32.129601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:32.129556 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:32.130139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:32.129638 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:33.128989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:33.128948 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:33.129185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:33.129014 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:34.128929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:34.128890 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:34.129394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:34.128962 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:35.129490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:35.129453 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:35.130005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:35.129528 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:36.129506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:36.129469 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:36.130069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:36.129528 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:37.129684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:37.129644 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:37.130191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:37.129706 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:38.129565 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:38.129532 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:38.129768 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:38.129588 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:39.128782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:39.128744 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:39.128988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:39.128821 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:39.128988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:39.128873 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:39.129567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:39.129546 5520 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"1ff892b288093ad0817ce15551c3e5de79c51d8380a54883765a4d85252e3c53"} pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" containerMessage="Container istio-proxy failed startup probe, will be restarted" Oct 31 03:51:39.129623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:39.129599 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" containerID="cri-o://1ff892b288093ad0817ce15551c3e5de79c51d8380a54883765a4d85252e3c53" gracePeriod=30 Oct 31 03:51:39.362847 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:51:39.362811 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:51:44.251756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:44.251716 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:51:44.252085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:44.251797 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:51:44.252085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:44.251831 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:51:44.731506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:44.731469 5520 generic.go:356] "Generic (PLEG): container finished" podID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerID="1ff892b288093ad0817ce15551c3e5de79c51d8380a54883765a4d85252e3c53" exitCode=0 Oct 31 03:51:44.731687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:44.731546 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" event={"ID":"8c92f1df-97ea-4f0f-8b10-3c04f109f33c","Type":"ContainerDied","Data":"1ff892b288093ad0817ce15551c3e5de79c51d8380a54883765a4d85252e3c53"} Oct 31 03:51:44.731687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:44.731608 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" event={"ID":"8c92f1df-97ea-4f0f-8b10-3c04f109f33c","Type":"ContainerStarted","Data":"e501340198ca63d97c2d278945a48721672830474f8b248e63919c70fcfc89dc"} Oct 31 03:51:45.131931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:45.129267 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:45.131931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:45.129392 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:45.131931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:45.129433 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:46.128782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:46.128745 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:46.129310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:46.128807 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:47.129282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:47.129246 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:47.129734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:47.129315 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:48.129160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:48.129122 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:48.129388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:48.129187 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:49.129154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:49.129118 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:49.129389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:49.129187 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:50.129240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:50.129168 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 31 03:51:50.129735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:50.129252 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 31 03:51:51.137709 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:51.137672 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:51.138787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:51.138767 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:51:51.139813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:51:51.139793 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:52:07.027011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:07.026969 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx"] Oct 31 03:52:07.033217 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:07.033175 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe127c9-hzggx"] Oct 31 03:52:07.923645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:07.923607 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bdf207-a347-4338-af98-61597014aa0e" path="/var/lib/kubelet/pods/f9bdf207-a347-4338-af98-61597014aa0e/volumes" Oct 31 03:52:09.364252 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:52:09.364187 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:52:17.952377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:17.952341 5520 scope.go:122] "RemoveContainer" containerID="559ebcd0d7b44737ad5dba80fae3519ed547a41872c33419a5d591e6c3222807" Oct 31 03:52:19.024014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:19.023978 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv"] Oct 31 03:52:19.027995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:19.027973 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d978df-thrjv"] Oct 31 03:52:19.922621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:19.922585 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fa4874-2d18-45d4-8f1e-0598742b2c0f" path="/var/lib/kubelet/pods/67fa4874-2d18-45d4-8f1e-0598742b2c0f/volumes" Oct 31 03:52:26.988495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:26.988417 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l"] Oct 31 03:52:26.988967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:26.988628 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" containerID="cri-o://3ca3eff0f565482397a354a9e33fc55a750985d57e8e0565184cc4d2a163c3fa" gracePeriod=30 Oct 31 03:52:30.320446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:30.320410 5520 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t5fpc_f454b995-5cf2-461d-b6f1-932bd4a45110/konnectivity-agent/0.log" limit=1024 Oct 31 03:52:32.215458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.215264 5520 generic.go:356] "Generic (PLEG): container finished" podID="af0b94dd-ab27-432b-aa69-5f551939d376" containerID="3ca3eff0f565482397a354a9e33fc55a750985d57e8e0565184cc4d2a163c3fa" exitCode=0 Oct 31 03:52:32.215458 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.215338 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" event={"ID":"af0b94dd-ab27-432b-aa69-5f551939d376","Type":"ContainerDied","Data":"3ca3eff0f565482397a354a9e33fc55a750985d57e8e0565184cc4d2a163c3fa"} Oct 31 03:52:32.315158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.315127 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:52:32.367452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.367414 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-socket\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.367713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.367685 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-socket" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:32.367861 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.367802 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0b94dd-ab27-432b-aa69-5f551939d376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0b94dd-ab27-432b-aa69-5f551939d376-istiod-ca-cert\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.367931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.367916 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-certs\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.367990 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.367948 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-credential-socket\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.368095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.368078 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-kube-api-access-ctww5\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-kube-api-access-ctww5\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.368262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.368131 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-envoy\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.368327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.368268 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-istio-token\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-istio-token\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.368383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.368346 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-data\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.368445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.368382 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af0b94dd-ab27-432b-aa69-5f551939d376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0b94dd-ab27-432b-aa69-5f551939d376-istio-podinfo\") pod \"af0b94dd-ab27-432b-aa69-5f551939d376\" (UID: \"af0b94dd-ab27-432b-aa69-5f551939d376\") " Oct 31 03:52:32.369096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.369078 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.370517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.370486 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-certs" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:32.370622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.370580 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-credential-socket" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:32.370747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.370724 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0b94dd-ab27-432b-aa69-5f551939d376-istiod-ca-cert" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:52:32.371042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.371016 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-data" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:32.372018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.371992 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af0b94dd-ab27-432b-aa69-5f551939d376-istio-podinfo" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:52:32.374418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.374387 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-istio-token" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:32.374522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.374467 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-envoy" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:52:32.375620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.375585 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-kube-api-access-ctww5" pod "af0b94dd-ab27-432b-aa69-5f551939d376" (UID: "af0b94dd-ab27-432b-aa69-5f551939d376"). InnerVolumeSpecName "kube-api-access-ctww5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:52:32.471211 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471176 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.471211 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471219 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.471443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471228 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0b94dd-ab27-432b-aa69-5f551939d376-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.471443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471238 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0b94dd-ab27-432b-aa69-5f551939d376-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.471443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471248 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.471443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471257 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.471443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471265 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctww5\" (UniqueName: \"kubernetes.io/projected/af0b94dd-ab27-432b-aa69-5f551939d376-kube-api-access-ctww5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:32.471443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:32.471274 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0b94dd-ab27-432b-aa69-5f551939d376-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:52:33.226235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.226174 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" Oct 31 03:52:33.226705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.226168 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" event={"ID":"af0b94dd-ab27-432b-aa69-5f551939d376","Type":"ContainerDied","Data":"b1077e1e346a5a28f28d69b251b665b6c1a72a314dd02ea10b03300e040e9dbe"} Oct 31 03:52:33.226705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.226354 5520 scope.go:122] "RemoveContainer" containerID="3ca3eff0f565482397a354a9e33fc55a750985d57e8e0565184cc4d2a163c3fa" Oct 31 03:52:33.261301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.261271 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l"] Oct 31 03:52:33.264292 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.264266 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l"] Oct 31 03:52:33.310284 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.310242 5520 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:33.310496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.310339 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-x7j6l" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:33.921184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:52:33.921150 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" path="/var/lib/kubelet/pods/af0b94dd-ab27-432b-aa69-5f551939d376/volumes" Oct 31 03:52:39.359753 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:52:39.359710 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:53:09.363469 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:53:09.363422 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:53:17.982939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:53:17.982897 5520 scope.go:122] "RemoveContainer" containerID="01cefda416b2c97fba45841443f7adfceff65668f52219d552c029b69ffb894a" Oct 31 03:53:17.994461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:53:17.994439 5520 scope.go:122] "RemoveContainer" containerID="490b7ec8d53f7db0013476bf05c5da64e02a293de8b12d770efc4a3d65e4e62c" Oct 31 03:53:39.372117 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:53:39.372064 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:54:07.980960 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:07.980878 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj"] Oct 31 03:54:07.981539 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:07.981102 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" containerID="cri-o://f13606988022784e984b0ad1a5e07c992fa1bf0f995ce6ad3ddb98488786ede4" gracePeriod=30 Oct 31 03:54:09.434286 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:54:09.434240 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:54:12.869846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:12.869809 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j"] Oct 31 03:54:12.870699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:12.870667 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" containerID="cri-o://8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c" gracePeriod=30 Oct 31 03:54:13.300961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.299535 5520 generic.go:356] "Generic (PLEG): container finished" podID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerID="f13606988022784e984b0ad1a5e07c992fa1bf0f995ce6ad3ddb98488786ede4" exitCode=0 Oct 31 03:54:13.301572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.301520 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" event={"ID":"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c","Type":"ContainerDied","Data":"f13606988022784e984b0ad1a5e07c992fa1bf0f995ce6ad3ddb98488786ede4"} Oct 31 03:54:13.418334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.418163 5520 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:54:13.418525 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.418366 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 03:54:13.623290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.623264 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:54:13.659117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659084 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-certs\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659219 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-podinfo\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659261 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-socket\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659299 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-data\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659462 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-envoy\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659469 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-certs" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:13.659639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659521 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-token\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-token\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659563 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istiod-ca-cert\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659597 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-socket" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:13.659639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659609 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-data" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:13.659838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659687 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-credential-socket\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.659838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659749 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-kube-api-access-bk2xt\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-kube-api-access-bk2xt\") pod \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\" (UID: \"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c\") " Oct 31 03:54:13.660303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.659989 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istiod-ca-cert" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:13.660303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.660251 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-credential-socket" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:13.660984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.660955 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.661118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.660990 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.661118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.661006 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.661118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.661021 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.661118 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.661034 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.662778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.662753 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-podinfo" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:13.662896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.662804 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-token" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:13.662896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.662825 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-kube-api-access-bk2xt" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "kube-api-access-bk2xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:13.663130 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.663106 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-envoy" pod "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" (UID: "9ec9a673-d7aa-4b7d-807e-d519c2a29e2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:13.766139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.766102 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.766139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.766142 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.766418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.766160 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk2xt\" (UniqueName: \"kubernetes.io/projected/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-kube-api-access-bk2xt\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:13.766418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:13.766176 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:14.311882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:14.311784 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" event={"ID":"9ec9a673-d7aa-4b7d-807e-d519c2a29e2c","Type":"ContainerDied","Data":"816d4ff9711500252b731782513adaf479b7de32dc2a762ddd723e707dd3001b"} Oct 31 03:54:14.311882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:14.311840 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj" Oct 31 03:54:14.311882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:14.311861 5520 scope.go:122] "RemoveContainer" containerID="f13606988022784e984b0ad1a5e07c992fa1bf0f995ce6ad3ddb98488786ede4" Oct 31 03:54:14.334213 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:14.334180 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj"] Oct 31 03:54:14.338511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:14.338483 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-swghj"] Oct 31 03:54:15.922437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:15.922401 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" path="/var/lib/kubelet/pods/9ec9a673-d7aa-4b7d-807e-d519c2a29e2c/volumes" Oct 31 03:54:18.038898 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.038869 5520 scope.go:122] "RemoveContainer" containerID="3f8844757b61bcb061933d4bed2b83b7f90e1ceb08aa0da2887d76dd57e348f8" Oct 31 03:54:18.263897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.263867 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:54:18.377468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.377429 5520 generic.go:356] "Generic (PLEG): container finished" podID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerID="8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c" exitCode=0 Oct 31 03:54:18.377647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.377475 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" Oct 31 03:54:18.377647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.377505 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" event={"ID":"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b","Type":"ContainerDied","Data":"8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c"} Oct 31 03:54:18.377647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.377565 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j" event={"ID":"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b","Type":"ContainerDied","Data":"44e872a11a560110bdc6386cf9ef386e01093faff0614a4031d687f2c0ee3dcf"} Oct 31 03:54:18.377647 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.377594 5520 scope.go:122] "RemoveContainer" containerID="8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c" Oct 31 03:54:18.389606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.389581 5520 scope.go:122] "RemoveContainer" containerID="8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c" Oct 31 03:54:18.389973 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:54:18.389943 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c\": container with ID starting with 8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c not found: ID does not exist" containerID="8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c" Oct 31 03:54:18.390085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.389985 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c"} err="failed to get container status \"8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c\": rpc error: code = NotFound desc = could not find container \"8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c\": container with ID starting with 8e2c2908d637c5f5e77a9ffaa40ba9c58d394ad5856c5563d61a247db1016e1c not found: ID does not exist" Oct 31 03:54:18.401510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.401482 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-credential-socket\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.401780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.401755 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-kube-api-access-mnbbq\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-kube-api-access-mnbbq\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.401864 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.401796 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-certs\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.401925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.401869 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-data\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.401994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.401978 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-podinfo\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.402081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.402068 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-socket\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.402184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.402171 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istiod-ca-cert\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.402267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.402228 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-token\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-token\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.402332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.402302 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-envoy\") pod \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\" (UID: \"e9e3fdd4-9152-4bb8-b98d-2445eb671f4b\") " Oct 31 03:54:18.406035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.406010 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-envoy" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.406527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.406469 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-data" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.406880 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.406855 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-certs" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.407081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.407061 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-credential-socket" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.410619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.410590 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-podinfo" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:18.410795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.410777 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-socket" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.411032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.411015 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istiod-ca-cert" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:18.418390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.418333 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-token" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:18.418390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.418366 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-kube-api-access-mnbbq" pod "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" (UID: "e9e3fdd4-9152-4bb8-b98d-2445eb671f4b"). InnerVolumeSpecName "kube-api-access-mnbbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:18.505178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505142 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505180 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnbbq\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-kube-api-access-mnbbq\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505244 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505258 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505274 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505288 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505301 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505314 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.505427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.505328 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.706017 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.705982 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j"] Oct 31 03:54:18.711908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:18.711877 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-xwr2j"] Oct 31 03:54:19.923496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:19.923457 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" path="/var/lib/kubelet/pods/e9e3fdd4-9152-4bb8-b98d-2445eb671f4b/volumes" Oct 31 03:54:23.607901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:23.607860 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5"] Oct 31 03:54:28.608406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:28.608357 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" podUID="24611e04-1978-4f47-b8d3-4234b0ce003e" containerName="payload-processing" containerID="cri-o://fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7" gracePeriod=25 Oct 31 03:54:28.936595 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:28.936562 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:54:29.022500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.022462 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24611e04-1978-4f47-b8d3-4234b0ce003e-kube-api-access-qn9wf\" (UniqueName: \"kubernetes.io/projected/24611e04-1978-4f47-b8d3-4234b0ce003e-kube-api-access-qn9wf\") pod \"24611e04-1978-4f47-b8d3-4234b0ce003e\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " Oct 31 03:54:29.022710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.022641 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24611e04-1978-4f47-b8d3-4234b0ce003e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24611e04-1978-4f47-b8d3-4234b0ce003e-plugins-config-volume\") pod \"24611e04-1978-4f47-b8d3-4234b0ce003e\" (UID: \"24611e04-1978-4f47-b8d3-4234b0ce003e\") " Oct 31 03:54:29.023075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.023046 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24611e04-1978-4f47-b8d3-4234b0ce003e-plugins-config-volume" pod "24611e04-1978-4f47-b8d3-4234b0ce003e" (UID: "24611e04-1978-4f47-b8d3-4234b0ce003e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:29.023886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.023857 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24611e04-1978-4f47-b8d3-4234b0ce003e-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:29.025021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.024997 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24611e04-1978-4f47-b8d3-4234b0ce003e-kube-api-access-qn9wf" pod "24611e04-1978-4f47-b8d3-4234b0ce003e" (UID: "24611e04-1978-4f47-b8d3-4234b0ce003e"). InnerVolumeSpecName "kube-api-access-qn9wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:29.125561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.125529 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn9wf\" (UniqueName: \"kubernetes.io/projected/24611e04-1978-4f47-b8d3-4234b0ce003e-kube-api-access-qn9wf\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:29.587618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.587575 5520 generic.go:356] "Generic (PLEG): container finished" podID="24611e04-1978-4f47-b8d3-4234b0ce003e" containerID="fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7" exitCode=0 Oct 31 03:54:29.587810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.587622 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" event={"ID":"24611e04-1978-4f47-b8d3-4234b0ce003e","Type":"ContainerDied","Data":"fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7"} Oct 31 03:54:29.587810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.587664 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" Oct 31 03:54:29.587810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.587703 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5" event={"ID":"24611e04-1978-4f47-b8d3-4234b0ce003e","Type":"ContainerDied","Data":"b0dcdd5988fb602355aa5695050326d7262b62069554079264f75cb981f8c5ae"} Oct 31 03:54:29.587810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.587736 5520 scope.go:122] "RemoveContainer" containerID="fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7" Oct 31 03:54:29.605257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.605141 5520 scope.go:122] "RemoveContainer" containerID="fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7" Oct 31 03:54:29.606587 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:54:29.606558 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7\": container with ID starting with fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7 not found: ID does not exist" containerID="fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7" Oct 31 03:54:29.606699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.606599 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7"} err="failed to get container status \"fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7\": rpc error: code = NotFound desc = could not find container \"fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7\": container with ID starting with fac3c24afde16c606bd12b852450731daea1511ad7ee190829d84c4713af63e7 not found: ID does not exist" Oct 31 03:54:29.619955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.619929 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5"] Oct 31 03:54:29.623906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.623882 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-46ebfe-6bd9fccb6c-m4wt5"] Oct 31 03:54:29.923381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:29.923307 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24611e04-1978-4f47-b8d3-4234b0ce003e" path="/var/lib/kubelet/pods/24611e04-1978-4f47-b8d3-4234b0ce003e/volumes" Oct 31 03:54:34.964850 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:34.964813 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45"] Oct 31 03:54:34.965408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:34.965029 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" podUID="dbde46c5-a7f8-4d6e-9452-ccda66a974e4" containerName="maas-api" containerID="cri-o://84bd4308c2950e33de37632e236598623ffa41628385fc7290f20950d5df9307" gracePeriod=30 Oct 31 03:54:35.736526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:35.736121 5520 generic.go:356] "Generic (PLEG): container finished" podID="dbde46c5-a7f8-4d6e-9452-ccda66a974e4" containerID="84bd4308c2950e33de37632e236598623ffa41628385fc7290f20950d5df9307" exitCode=0 Oct 31 03:54:35.736526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:35.736429 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" event={"ID":"dbde46c5-a7f8-4d6e-9452-ccda66a974e4","Type":"ContainerDied","Data":"84bd4308c2950e33de37632e236598623ffa41628385fc7290f20950d5df9307"} Oct 31 03:54:35.906476 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:35.906445 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:54:36.012367 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.012265 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-openshift-service-ca\") pod \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " Oct 31 03:54:36.012845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.012379 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-kube-api-access-grtdj\" (UniqueName: \"kubernetes.io/projected/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-kube-api-access-grtdj\") pod \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " Oct 31 03:54:36.012845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.012405 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-metrics-certs\") pod \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " Oct 31 03:54:36.012845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.012465 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-maas-api-tls\") pod \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\" (UID: \"dbde46c5-a7f8-4d6e-9452-ccda66a974e4\") " Oct 31 03:54:36.012845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.012712 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-openshift-service-ca" pod "dbde46c5-a7f8-4d6e-9452-ccda66a974e4" (UID: "dbde46c5-a7f8-4d6e-9452-ccda66a974e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:36.013519 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.013494 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.014962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.014930 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-maas-api-tls" pod "dbde46c5-a7f8-4d6e-9452-ccda66a974e4" (UID: "dbde46c5-a7f8-4d6e-9452-ccda66a974e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:36.015309 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.015281 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-metrics-certs" pod "dbde46c5-a7f8-4d6e-9452-ccda66a974e4" (UID: "dbde46c5-a7f8-4d6e-9452-ccda66a974e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:54:36.015405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.015352 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-kube-api-access-grtdj" pod "dbde46c5-a7f8-4d6e-9452-ccda66a974e4" (UID: "dbde46c5-a7f8-4d6e-9452-ccda66a974e4"). InnerVolumeSpecName "kube-api-access-grtdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:36.115343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.115304 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.115343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.115337 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grtdj\" (UniqueName: \"kubernetes.io/projected/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-kube-api-access-grtdj\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.115343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.115349 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbde46c5-a7f8-4d6e-9452-ccda66a974e4-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:36.747096 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.747069 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" Oct 31 03:54:36.747348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.747094 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45" event={"ID":"dbde46c5-a7f8-4d6e-9452-ccda66a974e4","Type":"ContainerDied","Data":"a4dc975ccfc58fc5c65daebb04bab4994b0418bfe4714ab57ac8a5586e13092f"} Oct 31 03:54:36.747348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.747156 5520 scope.go:122] "RemoveContainer" containerID="84bd4308c2950e33de37632e236598623ffa41628385fc7290f20950d5df9307" Oct 31 03:54:36.783890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.783848 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45"] Oct 31 03:54:36.803005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:36.801339 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46ebfe-5fb4874579-hqs45"] Oct 31 03:54:37.922223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:37.922172 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbde46c5-a7f8-4d6e-9452-ccda66a974e4" path="/var/lib/kubelet/pods/dbde46c5-a7f8-4d6e-9452-ccda66a974e4/volumes" Oct 31 03:54:39.361437 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:54:39.361398 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:54:42.386816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:42.386779 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995"] Oct 31 03:54:42.387730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:42.387682 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" containerID="cri-o://b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d" gracePeriod=30 Oct 31 03:54:42.416688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:42.415854 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq"] Oct 31 03:54:42.416688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:42.416087 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" containerID="cri-o://15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181" gracePeriod=30 Oct 31 03:54:47.865242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.865177 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:54:47.869913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.869877 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:54:47.964566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.964469 5520 generic.go:356] "Generic (PLEG): container finished" podID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerID="15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181" exitCode=0 Oct 31 03:54:47.964566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.964533 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" Oct 31 03:54:47.964566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.964547 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" event={"ID":"7e0d8f04-2f35-4581-a012-9e205b8ced0e","Type":"ContainerDied","Data":"15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181"} Oct 31 03:54:47.964841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.964617 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq" event={"ID":"7e0d8f04-2f35-4581-a012-9e205b8ced0e","Type":"ContainerDied","Data":"aac06d68a57a51a13861fef1542a0fe726b897edeb4f58bc9b9f78a44e5e85be"} Oct 31 03:54:47.964841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.964646 5520 scope.go:122] "RemoveContainer" containerID="15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181" Oct 31 03:54:47.967402 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.967371 5520 generic.go:356] "Generic (PLEG): container finished" podID="c69df727-93b9-498f-8ff0-07845d18ceea" containerID="b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d" exitCode=0 Oct 31 03:54:47.967689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.967666 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" event={"ID":"c69df727-93b9-498f-8ff0-07845d18ceea","Type":"ContainerDied","Data":"b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d"} Oct 31 03:54:47.967778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.967744 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" event={"ID":"c69df727-93b9-498f-8ff0-07845d18ceea","Type":"ContainerDied","Data":"4f783d7da9969353cb2be5b562f2d9b664ec24a44507f4116c04d701cecad24a"} Oct 31 03:54:47.967867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.967850 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995" Oct 31 03:54:47.972826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.972805 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-credential-socket\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.972944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.972838 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-credential-socket\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.972944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.972890 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-data\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.973075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.972974 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istiod-ca-cert\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973028 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-envoy\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.973075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973060 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-data\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973089 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-token\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-token\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973145 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-kube-api-access-rw59b\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-kube-api-access-rw59b\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973148 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-credential-socket" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.973253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973189 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-credential-socket" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.973486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973237 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-certs\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.973486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973380 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-socket\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973420 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-socket\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.973486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973454 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69df727-93b9-498f-8ff0-07845d18ceea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69df727-93b9-498f-8ff0-07845d18ceea-istiod-ca-cert\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.973486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973463 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-certs" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.973775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973607 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-envoy\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973629 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-certs\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973699 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-podinfo\") pod \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\" (UID: \"7e0d8f04-2f35-4581-a012-9e205b8ced0e\") " Oct 31 03:54:47.973775 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973744 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-socket" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.974063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.973825 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-data" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.974125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.974074 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69df727-93b9-498f-8ff0-07845d18ceea-istiod-ca-cert" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:47.974125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.974116 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-kube-api-access-z2gls\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-kube-api-access-z2gls\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.974289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.974122 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-data" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.974289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.974194 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-istio-token\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-istio-token\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.974289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.974243 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c69df727-93b9-498f-8ff0-07845d18ceea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69df727-93b9-498f-8ff0-07845d18ceea-istio-podinfo\") pod \"c69df727-93b9-498f-8ff0-07845d18ceea\" (UID: \"c69df727-93b9-498f-8ff0-07845d18ceea\") " Oct 31 03:54:47.974649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.974621 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istiod-ca-cert" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:47.975176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975130 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-certs" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.975354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975310 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-socket" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.975601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975570 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975625 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975641 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975656 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975670 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975682 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975697 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975711 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975723 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c69df727-93b9-498f-8ff0-07845d18ceea-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.975952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.975737 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.976540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.976518 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-envoy" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.977092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.976908 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-token" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:47.977682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.977656 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-envoy" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.977795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.977699 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c69df727-93b9-498f-8ff0-07845d18ceea-istio-podinfo" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:47.978117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.978093 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-istio-token" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:47.978350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.978326 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-podinfo" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:47.978522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.978507 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-kube-api-access-rw59b" pod "7e0d8f04-2f35-4581-a012-9e205b8ced0e" (UID: "7e0d8f04-2f35-4581-a012-9e205b8ced0e"). InnerVolumeSpecName "kube-api-access-rw59b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:47.979275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.979251 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-kube-api-access-z2gls" pod "c69df727-93b9-498f-8ff0-07845d18ceea" (UID: "c69df727-93b9-498f-8ff0-07845d18ceea"). InnerVolumeSpecName "kube-api-access-z2gls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:47.980562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.980543 5520 scope.go:122] "RemoveContainer" containerID="7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a" Oct 31 03:54:47.991592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.991573 5520 scope.go:122] "RemoveContainer" containerID="15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181" Oct 31 03:54:47.991989 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:54:47.991958 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181\": container with ID starting with 15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181 not found: ID does not exist" containerID="15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181" Oct 31 03:54:47.992101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.991996 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181"} err="failed to get container status \"15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181\": rpc error: code = NotFound desc = could not find container \"15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181\": container with ID starting with 15c1a7a86275f255e2afb5486bd28de84ada0b04f8005702b10f5934354eb181 not found: ID does not exist" Oct 31 03:54:47.992101 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.992018 5520 scope.go:122] "RemoveContainer" containerID="7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a" Oct 31 03:54:47.992423 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:54:47.992397 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a\": container with ID starting with 7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a not found: ID does not exist" containerID="7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a" Oct 31 03:54:47.992507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.992430 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a"} err="failed to get container status \"7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a\": rpc error: code = NotFound desc = could not find container \"7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a\": container with ID starting with 7e98cce30dc99f819c3b07307f477137aa78c8befb7f0ce843ed3a625f6ba08a not found: ID does not exist" Oct 31 03:54:47.992507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:47.992450 5520 scope.go:122] "RemoveContainer" containerID="b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d" Oct 31 03:54:48.003588 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.003570 5520 scope.go:122] "RemoveContainer" containerID="b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d" Oct 31 03:54:48.003872 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:54:48.003853 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d\": container with ID starting with b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d not found: ID does not exist" containerID="b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d" Oct 31 03:54:48.003935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.003881 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d"} err="failed to get container status \"b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d\": rpc error: code = NotFound desc = could not find container \"b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d\": container with ID starting with b788020e7dd0e3be0bb3bc2d4f30a234ff39c786c99b8324a6896a4b18d26e7d not found: ID does not exist" Oct 31 03:54:48.077811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077781 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c69df727-93b9-498f-8ff0-07845d18ceea-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.077811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077812 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.078024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077844 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw59b\" (UniqueName: \"kubernetes.io/projected/7e0d8f04-2f35-4581-a012-9e205b8ced0e-kube-api-access-rw59b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.078024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077870 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.078024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077888 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0d8f04-2f35-4581-a012-9e205b8ced0e-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.078024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077903 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2gls\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-kube-api-access-z2gls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.078024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077918 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c69df727-93b9-498f-8ff0-07845d18ceea-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.078024 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.077928 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c69df727-93b9-498f-8ff0-07845d18ceea-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.300357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.300313 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq"] Oct 31 03:54:48.358041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.358013 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5cntq"] Oct 31 03:54:48.374492 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.374454 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995"] Oct 31 03:54:48.379189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:48.379160 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-ld995"] Oct 31 03:54:49.950820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:49.950782 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" path="/var/lib/kubelet/pods/7e0d8f04-2f35-4581-a012-9e205b8ced0e/volumes" Oct 31 03:54:49.951616 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:54:49.951594 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" path="/var/lib/kubelet/pods/c69df727-93b9-498f-8ff0-07845d18ceea/volumes" Oct 31 03:55:09.364057 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:09.364020 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:55:18.116225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:18.115623 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s"] Oct 31 03:55:18.979188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:18.979148 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw"] Oct 31 03:55:22.161604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:22.161570 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc"] Oct 31 03:55:22.162092 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:22.161805 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" containerID="cri-o://f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104" gracePeriod=30 Oct 31 03:55:22.464036 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:22.463994 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx"] Oct 31 03:55:22.464316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:22.464291 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" containerID="cri-o://9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541" gracePeriod=30 Oct 31 03:55:23.116678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.116621 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" podUID="a4b73f30-4c6f-4f53-a679-84694df92392" containerName="payload-processing" containerID="cri-o://a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237" gracePeriod=25 Oct 31 03:55:23.681746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.669365 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:55:23.790228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.790085 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b73f30-4c6f-4f53-a679-84694df92392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b73f30-4c6f-4f53-a679-84694df92392-plugins-config-volume\") pod \"a4b73f30-4c6f-4f53-a679-84694df92392\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " Oct 31 03:55:23.794223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.790511 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b73f30-4c6f-4f53-a679-84694df92392-kube-api-access-ph24p\" (UniqueName: \"kubernetes.io/projected/a4b73f30-4c6f-4f53-a679-84694df92392-kube-api-access-ph24p\") pod \"a4b73f30-4c6f-4f53-a679-84694df92392\" (UID: \"a4b73f30-4c6f-4f53-a679-84694df92392\") " Oct 31 03:55:23.794223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.792002 5520 generic.go:356] "Generic (PLEG): container finished" podID="a4b73f30-4c6f-4f53-a679-84694df92392" containerID="a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237" exitCode=0 Oct 31 03:55:23.794223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.792097 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" event={"ID":"a4b73f30-4c6f-4f53-a679-84694df92392","Type":"ContainerDied","Data":"a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237"} Oct 31 03:55:23.794223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.792173 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" event={"ID":"a4b73f30-4c6f-4f53-a679-84694df92392","Type":"ContainerDied","Data":"41422419aea822d37f7441ff3361e4f4e2d69ca537fd8377ba4d5ed0a8187b0b"} Oct 31 03:55:23.797896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.796420 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b73f30-4c6f-4f53-a679-84694df92392-plugins-config-volume" pod "a4b73f30-4c6f-4f53-a679-84694df92392" (UID: "a4b73f30-4c6f-4f53-a679-84694df92392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:23.797896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.796530 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s" Oct 31 03:55:23.849324 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.847756 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b73f30-4c6f-4f53-a679-84694df92392-kube-api-access-ph24p" pod "a4b73f30-4c6f-4f53-a679-84694df92392" (UID: "a4b73f30-4c6f-4f53-a679-84694df92392"). InnerVolumeSpecName "kube-api-access-ph24p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:23.856448 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.856229 5520 scope.go:122] "RemoveContainer" containerID="a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237" Oct 31 03:55:23.913761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.913718 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph24p\" (UniqueName: \"kubernetes.io/projected/a4b73f30-4c6f-4f53-a679-84694df92392-kube-api-access-ph24p\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:23.913761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.913757 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4b73f30-4c6f-4f53-a679-84694df92392-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:23.984976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:23.984886 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" podUID="50b13be8-fc9d-4cac-9193-658b2bb30eae" containerName="payload-processing" containerID="cri-o://a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4" gracePeriod=25 Oct 31 03:55:24.089859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.089839 5520 scope.go:122] "RemoveContainer" containerID="a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237" Oct 31 03:55:24.090285 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:24.090258 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237\": container with ID starting with a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237 not found: ID does not exist" containerID="a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237" Oct 31 03:55:24.090460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.090296 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237"} err="failed to get container status \"a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237\": rpc error: code = NotFound desc = could not find container \"a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237\": container with ID starting with a384c27d55084f6ef326e4bb17215873214d02df0128c2300cf4c32b356f8237 not found: ID does not exist" Oct 31 03:55:24.125340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.124021 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s"] Oct 31 03:55:24.129480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.129439 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d4145b-78f74d699c-whc9s"] Oct 31 03:55:24.324941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.324911 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:55:24.527432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.527403 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b13be8-fc9d-4cac-9193-658b2bb30eae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b13be8-fc9d-4cac-9193-658b2bb30eae-plugins-config-volume\") pod \"50b13be8-fc9d-4cac-9193-658b2bb30eae\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " Oct 31 03:55:24.527624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.527555 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b13be8-fc9d-4cac-9193-658b2bb30eae-kube-api-access-glqf7\" (UniqueName: \"kubernetes.io/projected/50b13be8-fc9d-4cac-9193-658b2bb30eae-kube-api-access-glqf7\") pod \"50b13be8-fc9d-4cac-9193-658b2bb30eae\" (UID: \"50b13be8-fc9d-4cac-9193-658b2bb30eae\") " Oct 31 03:55:24.527840 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.527813 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b13be8-fc9d-4cac-9193-658b2bb30eae-plugins-config-volume" pod "50b13be8-fc9d-4cac-9193-658b2bb30eae" (UID: "50b13be8-fc9d-4cac-9193-658b2bb30eae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:24.528145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.528127 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b13be8-fc9d-4cac-9193-658b2bb30eae-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:24.530074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.530047 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b13be8-fc9d-4cac-9193-658b2bb30eae-kube-api-access-glqf7" pod "50b13be8-fc9d-4cac-9193-658b2bb30eae" (UID: "50b13be8-fc9d-4cac-9193-658b2bb30eae"). InnerVolumeSpecName "kube-api-access-glqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:24.561531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.561268 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx"] Oct 31 03:55:24.561713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.561576 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" podUID="70694229-8fa5-4e3f-af7f-f875faba310b" containerName="maas-api" containerID="cri-o://a75a0a11a273bb31127ff8de1aef0416a664c8018ffefb998969f92a80aecbcb" gracePeriod=30 Oct 31 03:55:24.629810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:24.629777 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqf7\" (UniqueName: \"kubernetes.io/projected/50b13be8-fc9d-4cac-9193-658b2bb30eae-kube-api-access-glqf7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:25.183278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.174103 5520 generic.go:356] "Generic (PLEG): container finished" podID="70694229-8fa5-4e3f-af7f-f875faba310b" containerID="a75a0a11a273bb31127ff8de1aef0416a664c8018ffefb998969f92a80aecbcb" exitCode=0 Oct 31 03:55:25.183278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.174225 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" event={"ID":"70694229-8fa5-4e3f-af7f-f875faba310b","Type":"ContainerDied","Data":"a75a0a11a273bb31127ff8de1aef0416a664c8018ffefb998969f92a80aecbcb"} Oct 31 03:55:25.183278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.176523 5520 generic.go:356] "Generic (PLEG): container finished" podID="50b13be8-fc9d-4cac-9193-658b2bb30eae" containerID="a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4" exitCode=0 Oct 31 03:55:25.183278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.176638 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" event={"ID":"50b13be8-fc9d-4cac-9193-658b2bb30eae","Type":"ContainerDied","Data":"a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4"} Oct 31 03:55:25.183278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.176699 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" event={"ID":"50b13be8-fc9d-4cac-9193-658b2bb30eae","Type":"ContainerDied","Data":"04b2375475aea7423483d0adcf6825a23b9576a8c2ce683ae8a46a3f460abae7"} Oct 31 03:55:25.183278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.176744 5520 scope.go:122] "RemoveContainer" containerID="a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4" Oct 31 03:55:25.183278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.176877 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw" Oct 31 03:55:25.490820 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.490268 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw"] Oct 31 03:55:25.650239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.649954 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-afb8be-7f57c4db59-xnqsw"] Oct 31 03:55:25.676667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.676628 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554"] Oct 31 03:55:25.701822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.691565 5520 scope.go:122] "RemoveContainer" containerID="a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4" Oct 31 03:55:25.701822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.697579 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:55:25.701822 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:25.697694 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4\": container with ID starting with a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4 not found: ID does not exist" containerID="a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4" Oct 31 03:55:25.701822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.697727 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4"} err="failed to get container status \"a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4\": rpc error: code = NotFound desc = could not find container \"a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4\": container with ID starting with a2929ba4c4438cab801575d32191a784e6c3a6d38992c1d53ee795278a5d6ef4 not found: ID does not exist" Oct 31 03:55:25.772451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.768729 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-metrics-certs\") pod \"70694229-8fa5-4e3f-af7f-f875faba310b\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " Oct 31 03:55:25.772451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.768805 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-maas-api-tls\") pod \"70694229-8fa5-4e3f-af7f-f875faba310b\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " Oct 31 03:55:25.772451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.769022 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70694229-8fa5-4e3f-af7f-f875faba310b-kube-api-access-sk4hf\" (UniqueName: \"kubernetes.io/projected/70694229-8fa5-4e3f-af7f-f875faba310b-kube-api-access-sk4hf\") pod \"70694229-8fa5-4e3f-af7f-f875faba310b\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " Oct 31 03:55:25.772451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.769088 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70694229-8fa5-4e3f-af7f-f875faba310b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70694229-8fa5-4e3f-af7f-f875faba310b-openshift-service-ca\") pod \"70694229-8fa5-4e3f-af7f-f875faba310b\" (UID: \"70694229-8fa5-4e3f-af7f-f875faba310b\") " Oct 31 03:55:25.772451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.771481 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70694229-8fa5-4e3f-af7f-f875faba310b-openshift-service-ca" pod "70694229-8fa5-4e3f-af7f-f875faba310b" (UID: "70694229-8fa5-4e3f-af7f-f875faba310b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:25.784603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.783328 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-maas-api-tls" pod "70694229-8fa5-4e3f-af7f-f875faba310b" (UID: "70694229-8fa5-4e3f-af7f-f875faba310b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:25.784603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.783465 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-metrics-certs" pod "70694229-8fa5-4e3f-af7f-f875faba310b" (UID: "70694229-8fa5-4e3f-af7f-f875faba310b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:25.790841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.786390 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70694229-8fa5-4e3f-af7f-f875faba310b-kube-api-access-sk4hf" pod "70694229-8fa5-4e3f-af7f-f875faba310b" (UID: "70694229-8fa5-4e3f-af7f-f875faba310b"). InnerVolumeSpecName "kube-api-access-sk4hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:25.896230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.893112 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk4hf\" (UniqueName: \"kubernetes.io/projected/70694229-8fa5-4e3f-af7f-f875faba310b-kube-api-access-sk4hf\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:25.896230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.893150 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70694229-8fa5-4e3f-af7f-f875faba310b-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:25.896230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.893164 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:25.896230 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.893178 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70694229-8fa5-4e3f-af7f-f875faba310b-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:25.999713 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:25.990922 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b13be8-fc9d-4cac-9193-658b2bb30eae" path="/var/lib/kubelet/pods/50b13be8-fc9d-4cac-9193-658b2bb30eae/volumes" Oct 31 03:55:26.006021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.005797 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b73f30-4c6f-4f53-a679-84694df92392" path="/var/lib/kubelet/pods/a4b73f30-4c6f-4f53-a679-84694df92392/volumes" Oct 31 03:55:26.208277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.207068 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" event={"ID":"70694229-8fa5-4e3f-af7f-f875faba310b","Type":"ContainerDied","Data":"4f8aad11d39dcfc9cf4fd4e88ea7df753a0771905af6b70704676f1897e2ef3e"} Oct 31 03:55:26.208277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.207234 5520 scope.go:122] "RemoveContainer" containerID="a75a0a11a273bb31127ff8de1aef0416a664c8018ffefb998969f92a80aecbcb" Oct 31 03:55:26.208277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.207433 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx" Oct 31 03:55:26.272399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.272362 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx"] Oct 31 03:55:26.272660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.272645 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d4145b-bddc578f4-ddrhx"] Oct 31 03:55:26.446342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.446305 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n"] Oct 31 03:55:26.452463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:26.452399 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6"] Oct 31 03:55:27.544117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.544088 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:55:27.626772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.626731 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-certs\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.626978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.626823 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49484854-701e-49f5-9aa0-85e2338de33c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49484854-701e-49f5-9aa0-85e2338de33c-istiod-ca-cert\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.626978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.626921 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-data\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.627116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.627002 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-envoy\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.627116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.627037 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-kube-api-access-bt49k\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-kube-api-access-bt49k\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.627264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.627145 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-socket\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.627264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.627165 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-credential-socket\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.627264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.627196 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49484854-701e-49f5-9aa0-85e2338de33c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49484854-701e-49f5-9aa0-85e2338de33c-istio-podinfo\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.627264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.627235 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-istio-token\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-istio-token\") pod \"49484854-701e-49f5-9aa0-85e2338de33c\" (UID: \"49484854-701e-49f5-9aa0-85e2338de33c\") " Oct 31 03:55:27.628087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.628021 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49484854-701e-49f5-9aa0-85e2338de33c-istiod-ca-cert" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:27.628300 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.628276 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-socket" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.628605 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.628582 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-credential-socket" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.628787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.628765 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-certs" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.628787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.628778 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-data" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.634495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.630410 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-envoy" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.640920 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.640879 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-kube-api-access-bt49k" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "kube-api-access-bt49k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:27.643285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.643257 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49484854-701e-49f5-9aa0-85e2338de33c-istio-podinfo" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:55:27.648313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.648286 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-istio-token" pod "49484854-701e-49f5-9aa0-85e2338de33c" (UID: "49484854-701e-49f5-9aa0-85e2338de33c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.729897 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.729937 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49484854-701e-49f5-9aa0-85e2338de33c-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.729953 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.729967 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.729982 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt49k\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-kube-api-access-bt49k\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.729999 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.730013 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49484854-701e-49f5-9aa0-85e2338de33c-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.730027 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49484854-701e-49f5-9aa0-85e2338de33c-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.730462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.730041 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49484854-701e-49f5-9aa0-85e2338de33c-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.788994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.788963 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:55:27.921224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.921174 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70694229-8fa5-4e3f-af7f-f875faba310b" path="/var/lib/kubelet/pods/70694229-8fa5-4e3f-af7f-f875faba310b/volumes" Oct 31 03:55:27.933996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.933967 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-data\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.934282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934256 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-data" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934450 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-certs\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934521 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istiod-ca-cert\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934660 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-podinfo\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934699 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-envoy\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934767 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-credential-socket\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934826 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-socket\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934936 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-kube-api-access-qwcz4\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-kube-api-access-qwcz4\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.934970 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-token\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-token\") pod \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\" (UID: \"a8d58c9a-ac48-41b7-bf68-c54fbea4531e\") " Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.935716 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.958401 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-credential-socket" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.958679 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-socket" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.974078 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-certs" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.975254 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:27.974502 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istiod-ca-cert" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:28.026531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.019565 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-envoy" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:28.026531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.021041 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-podinfo" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:55:28.026531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.026411 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-token" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:28.026969 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.026934 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-kube-api-access-qwcz4" pod "a8d58c9a-ac48-41b7-bf68-c54fbea4531e" (UID: "a8d58c9a-ac48-41b7-bf68-c54fbea4531e"). InnerVolumeSpecName "kube-api-access-qwcz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038570 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038604 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038619 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038633 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038651 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcz4\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-kube-api-access-qwcz4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038666 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038679 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.040224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.038696 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d58c9a-ac48-41b7-bf68-c54fbea4531e-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:28.239475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.239442 5520 generic.go:356] "Generic (PLEG): container finished" podID="49484854-701e-49f5-9aa0-85e2338de33c" containerID="f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104" exitCode=0 Oct 31 03:55:28.239668 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.239545 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" Oct 31 03:55:28.239668 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.239588 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" event={"ID":"49484854-701e-49f5-9aa0-85e2338de33c","Type":"ContainerDied","Data":"f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104"} Oct 31 03:55:28.239668 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.239649 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc" event={"ID":"49484854-701e-49f5-9aa0-85e2338de33c","Type":"ContainerDied","Data":"e3c3c00c9dca9c54825ecee9c4c21bbd51826b37b83603d19e6f7312f323e368"} Oct 31 03:55:28.239851 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.239679 5520 scope.go:122] "RemoveContainer" containerID="f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104" Oct 31 03:55:28.241583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.241450 5520 generic.go:356] "Generic (PLEG): container finished" podID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerID="9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541" exitCode=0 Oct 31 03:55:28.241583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.241499 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" event={"ID":"a8d58c9a-ac48-41b7-bf68-c54fbea4531e","Type":"ContainerDied","Data":"9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541"} Oct 31 03:55:28.241583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.241562 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" event={"ID":"a8d58c9a-ac48-41b7-bf68-c54fbea4531e","Type":"ContainerDied","Data":"076557dafa987a70aff1d969aad1a0b1390f8be8ab9e5051dbee36f627810b1e"} Oct 31 03:55:28.241583 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.241506 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx" Oct 31 03:55:28.281641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.281555 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc"] Oct 31 03:55:28.288506 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.287701 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-bb2rc"] Oct 31 03:55:28.289239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.289187 5520 scope.go:122] "RemoveContainer" containerID="f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104" Oct 31 03:55:28.289727 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:28.289695 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104\": container with ID starting with f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104 not found: ID does not exist" containerID="f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104" Oct 31 03:55:28.289872 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.289737 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104"} err="failed to get container status \"f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104\": rpc error: code = NotFound desc = could not find container \"f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104\": container with ID starting with f45d0f09c9e6592526e8433aaed4385573cd21a03ad751b7c17098fb26b38104 not found: ID does not exist" Oct 31 03:55:28.289872 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.289762 5520 scope.go:122] "RemoveContainer" containerID="9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541" Oct 31 03:55:28.320043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.319993 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx"] Oct 31 03:55:28.327589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.327516 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-mhnmx"] Oct 31 03:55:28.336661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.336634 5520 scope.go:122] "RemoveContainer" containerID="9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541" Oct 31 03:55:28.337585 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:28.337153 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541\": container with ID starting with 9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541 not found: ID does not exist" containerID="9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541" Oct 31 03:55:28.337585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:28.337264 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541"} err="failed to get container status \"9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541\": rpc error: code = NotFound desc = could not find container \"9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541\": container with ID starting with 9243d44fe2cccbaab4b700ee9e710ce08c1e6302c3d4f6e001eb658c1c13f541 not found: ID does not exist" Oct 31 03:55:29.919949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:29.919917 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49484854-701e-49f5-9aa0-85e2338de33c" path="/var/lib/kubelet/pods/49484854-701e-49f5-9aa0-85e2338de33c/volumes" Oct 31 03:55:29.920761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:29.920738 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" path="/var/lib/kubelet/pods/a8d58c9a-ac48-41b7-bf68-c54fbea4531e/volumes" Oct 31 03:55:31.447794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:31.447753 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" podUID="52dbfac7-97ba-468a-b635-ba2a7faf6ef1" containerName="payload-processing" containerID="cri-o://3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb" gracePeriod=25 Oct 31 03:55:31.453290 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:31.453258 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" podUID="bd63b361-66b8-4cc5-a8db-619eaf42496b" containerName="payload-pre-processing" containerID="cri-o://f8b29037677fd634a6e3a77138ac18772ce0b21e24fd5600e4bf6d066dec8be5" gracePeriod=25 Oct 31 03:55:31.687346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:31.684647 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s"] Oct 31 03:55:32.039915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.019598 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp"] Oct 31 03:55:32.329278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.313946 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf"] Oct 31 03:55:32.333993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.332427 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:55:32.340407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.340377 5520 generic.go:356] "Generic (PLEG): container finished" podID="bd63b361-66b8-4cc5-a8db-619eaf42496b" containerID="f8b29037677fd634a6e3a77138ac18772ce0b21e24fd5600e4bf6d066dec8be5" exitCode=0 Oct 31 03:55:32.340615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.340592 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" event={"ID":"bd63b361-66b8-4cc5-a8db-619eaf42496b","Type":"ContainerDied","Data":"f8b29037677fd634a6e3a77138ac18772ce0b21e24fd5600e4bf6d066dec8be5"} Oct 31 03:55:32.349238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.348534 5520 generic.go:356] "Generic (PLEG): container finished" podID="52dbfac7-97ba-468a-b635-ba2a7faf6ef1" containerID="3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb" exitCode=0 Oct 31 03:55:32.349238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.348653 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" event={"ID":"52dbfac7-97ba-468a-b635-ba2a7faf6ef1","Type":"ContainerDied","Data":"3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb"} Oct 31 03:55:32.349238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.348697 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" event={"ID":"52dbfac7-97ba-468a-b635-ba2a7faf6ef1","Type":"ContainerDied","Data":"09dddb68fcf1edf5ed61f93e7e167f2c6d54f3ef9c9a419519750012571a9e0b"} Oct 31 03:55:32.349238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.348726 5520 scope.go:122] "RemoveContainer" containerID="3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb" Oct 31 03:55:32.349238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.348849 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n" Oct 31 03:55:32.358086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.357995 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:55:32.448351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.436403 5520 scope.go:122] "RemoveContainer" containerID="3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb" Oct 31 03:55:32.480651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.470305 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-plugins-config-volume\") pod \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " Oct 31 03:55:32.480651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.470505 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd63b361-66b8-4cc5-a8db-619eaf42496b-kube-api-access-p75z2\" (UniqueName: \"kubernetes.io/projected/bd63b361-66b8-4cc5-a8db-619eaf42496b-kube-api-access-p75z2\") pod \"bd63b361-66b8-4cc5-a8db-619eaf42496b\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " Oct 31 03:55:32.482015 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:32.481966 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb\": container with ID starting with 3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb not found: ID does not exist" containerID="3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb" Oct 31 03:55:32.498451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.493266 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb"} err="failed to get container status \"3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb\": rpc error: code = NotFound desc = could not find container \"3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb\": container with ID starting with 3ee90c998f12abd5def175c1025f33b562638c43610cbe1961b481ac91dde0bb not found: ID does not exist" Oct 31 03:55:32.498451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.486872 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd63b361-66b8-4cc5-a8db-619eaf42496b-plugins-config-volume" pod "bd63b361-66b8-4cc5-a8db-619eaf42496b" (UID: "bd63b361-66b8-4cc5-a8db-619eaf42496b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:32.498451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.487449 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-plugins-config-volume" pod "52dbfac7-97ba-468a-b635-ba2a7faf6ef1" (UID: "52dbfac7-97ba-468a-b635-ba2a7faf6ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:32.498451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.486492 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd63b361-66b8-4cc5-a8db-619eaf42496b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd63b361-66b8-4cc5-a8db-619eaf42496b-plugins-config-volume\") pod \"bd63b361-66b8-4cc5-a8db-619eaf42496b\" (UID: \"bd63b361-66b8-4cc5-a8db-619eaf42496b\") " Oct 31 03:55:32.498451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.493439 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-kube-api-access-8wt97\" (UniqueName: \"kubernetes.io/projected/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-kube-api-access-8wt97\") pod \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\" (UID: \"52dbfac7-97ba-468a-b635-ba2a7faf6ef1\") " Oct 31 03:55:32.498451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.494286 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd63b361-66b8-4cc5-a8db-619eaf42496b-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:32.498451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.494309 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:32.562145 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.556687 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-kube-api-access-8wt97" pod "52dbfac7-97ba-468a-b635-ba2a7faf6ef1" (UID: "52dbfac7-97ba-468a-b635-ba2a7faf6ef1"). InnerVolumeSpecName "kube-api-access-8wt97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:32.596270 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.595624 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wt97\" (UniqueName: \"kubernetes.io/projected/52dbfac7-97ba-468a-b635-ba2a7faf6ef1-kube-api-access-8wt97\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:32.610877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.597521 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz"] Oct 31 03:55:32.861058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.860962 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd63b361-66b8-4cc5-a8db-619eaf42496b-kube-api-access-p75z2" pod "bd63b361-66b8-4cc5-a8db-619eaf42496b" (UID: "bd63b361-66b8-4cc5-a8db-619eaf42496b"). InnerVolumeSpecName "kube-api-access-p75z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:32.901562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.901517 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n"] Oct 31 03:55:32.909040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.908930 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b81d65-57c9f68b89-xbn6n"] Oct 31 03:55:32.933172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:32.932680 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p75z2\" (UniqueName: \"kubernetes.io/projected/bd63b361-66b8-4cc5-a8db-619eaf42496b-kube-api-access-p75z2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:33.148807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.148719 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2"] Oct 31 03:55:33.433013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.432979 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" event={"ID":"bd63b361-66b8-4cc5-a8db-619eaf42496b","Type":"ContainerDied","Data":"d64d2df9d2c64bce64b99732cadd51022b4635166270bd9e6239387be5c8a0e8"} Oct 31 03:55:33.434007 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.433988 5520 scope.go:122] "RemoveContainer" containerID="f8b29037677fd634a6e3a77138ac18772ce0b21e24fd5600e4bf6d066dec8be5" Oct 31 03:55:33.434279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.433912 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6" Oct 31 03:55:33.442464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.442434 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254"] Oct 31 03:55:33.496416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.496383 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6"] Oct 31 03:55:33.501468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.501356 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b81d65-545775d8c5-hgkz6"] Oct 31 03:55:33.837349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:33.837253 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6"] Oct 31 03:55:34.284314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:34.273645 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52dbfac7-97ba-468a-b635-ba2a7faf6ef1" path="/var/lib/kubelet/pods/52dbfac7-97ba-468a-b635-ba2a7faf6ef1/volumes" Oct 31 03:55:34.284314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:34.274683 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd63b361-66b8-4cc5-a8db-619eaf42496b" path="/var/lib/kubelet/pods/bd63b361-66b8-4cc5-a8db-619eaf42496b/volumes" Oct 31 03:55:34.284314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:34.275354 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx"] Oct 31 03:55:39.386690 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:39.386650 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:55:40.717308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:40.717262 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" podUID="e86477ac-83fb-4266-a83f-560f10bdb64c" containerName="main" containerID="cri-o://e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c" gracePeriod=45 Oct 31 03:55:41.094330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.092408 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:55:41.172222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.172153 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-dshm\") pod \"e86477ac-83fb-4266-a83f-560f10bdb64c\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " Oct 31 03:55:41.172409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.172364 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86477ac-83fb-4266-a83f-560f10bdb64c-kube-api-access-d2sgn\" (UniqueName: \"kubernetes.io/projected/e86477ac-83fb-4266-a83f-560f10bdb64c-kube-api-access-d2sgn\") pod \"e86477ac-83fb-4266-a83f-560f10bdb64c\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " Oct 31 03:55:41.172571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.172550 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-model-cache\") pod \"e86477ac-83fb-4266-a83f-560f10bdb64c\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " Oct 31 03:55:41.172634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.172610 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e86477ac-83fb-4266-a83f-560f10bdb64c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e86477ac-83fb-4266-a83f-560f10bdb64c-tls-certs\") pod \"e86477ac-83fb-4266-a83f-560f10bdb64c\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " Oct 31 03:55:41.172689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.172664 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-home\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-home\") pod \"e86477ac-83fb-4266-a83f-560f10bdb64c\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " Oct 31 03:55:41.172844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.172825 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-tmp-dir\") pod \"e86477ac-83fb-4266-a83f-560f10bdb64c\" (UID: \"e86477ac-83fb-4266-a83f-560f10bdb64c\") " Oct 31 03:55:41.173377 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.173223 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-home" pod "e86477ac-83fb-4266-a83f-560f10bdb64c" (UID: "e86477ac-83fb-4266-a83f-560f10bdb64c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:41.173491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.173403 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-tmp-dir" pod "e86477ac-83fb-4266-a83f-560f10bdb64c" (UID: "e86477ac-83fb-4266-a83f-560f10bdb64c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:41.173604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.173581 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-model-cache" pod "e86477ac-83fb-4266-a83f-560f10bdb64c" (UID: "e86477ac-83fb-4266-a83f-560f10bdb64c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:41.174022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.173980 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:41.174022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.173994 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:41.174022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.174004 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:41.175352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.175282 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-dshm" pod "e86477ac-83fb-4266-a83f-560f10bdb64c" (UID: "e86477ac-83fb-4266-a83f-560f10bdb64c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:41.175460 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.175299 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86477ac-83fb-4266-a83f-560f10bdb64c-tls-certs" pod "e86477ac-83fb-4266-a83f-560f10bdb64c" (UID: "e86477ac-83fb-4266-a83f-560f10bdb64c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:41.176571 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.176549 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86477ac-83fb-4266-a83f-560f10bdb64c-kube-api-access-d2sgn" pod "e86477ac-83fb-4266-a83f-560f10bdb64c" (UID: "e86477ac-83fb-4266-a83f-560f10bdb64c"). InnerVolumeSpecName "kube-api-access-d2sgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:41.276091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.276002 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2sgn\" (UniqueName: \"kubernetes.io/projected/e86477ac-83fb-4266-a83f-560f10bdb64c-kube-api-access-d2sgn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:41.276091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.276035 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86477ac-83fb-4266-a83f-560f10bdb64c-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:41.276091 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.276050 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86477ac-83fb-4266-a83f-560f10bdb64c-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:41.843286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.843253 5520 generic.go:356] "Generic (PLEG): container finished" podID="e86477ac-83fb-4266-a83f-560f10bdb64c" containerID="e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c" exitCode=0 Oct 31 03:55:41.843757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.843301 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" Oct 31 03:55:41.843757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.843327 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" event={"ID":"e86477ac-83fb-4266-a83f-560f10bdb64c","Type":"ContainerDied","Data":"e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c"} Oct 31 03:55:41.843757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.843369 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554" event={"ID":"e86477ac-83fb-4266-a83f-560f10bdb64c","Type":"ContainerDied","Data":"5f4fcd6dd98e19a92036dc191a6b5ee84962950b527f94de5f510ecf75150336"} Oct 31 03:55:41.843757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.843388 5520 scope.go:122] "RemoveContainer" containerID="e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c" Oct 31 03:55:41.854901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.854881 5520 scope.go:122] "RemoveContainer" containerID="e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c" Oct 31 03:55:41.855257 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:41.855231 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c\": container with ID starting with e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c not found: ID does not exist" containerID="e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c" Oct 31 03:55:41.855360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.855262 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c"} err="failed to get container status \"e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c\": rpc error: code = NotFound desc = could not find container \"e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c\": container with ID starting with e915916e7fbe32d582b4baf24f03cf5619e6163ef98943278b4cb788b13c515c not found: ID does not exist" Oct 31 03:55:41.866968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.866944 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554"] Oct 31 03:55:41.871174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.871151 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b8fd22/auth-test-model-b8fd22-kserve-78f688b749-ph554"] Oct 31 03:55:41.933662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:41.933617 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86477ac-83fb-4266-a83f-560f10bdb64c" path="/var/lib/kubelet/pods/e86477ac-83fb-4266-a83f-560f10bdb64c/volumes" Oct 31 03:55:44.676889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:44.674985 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh"] Oct 31 03:55:44.676889 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:44.675252 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" podUID="315c9d44-5d61-47a5-8bf7-85172dea8af2" containerName="maas-api" containerID="cri-o://10fb61703c5ba4849b167ea977b23997d6e1c8d35369e2ad2109033b91bcd3f1" gracePeriod=30 Oct 31 03:55:45.114229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.062328 5520 generic.go:356] "Generic (PLEG): container finished" podID="315c9d44-5d61-47a5-8bf7-85172dea8af2" containerID="10fb61703c5ba4849b167ea977b23997d6e1c8d35369e2ad2109033b91bcd3f1" exitCode=0 Oct 31 03:55:45.114229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.062494 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" event={"ID":"315c9d44-5d61-47a5-8bf7-85172dea8af2","Type":"ContainerDied","Data":"10fb61703c5ba4849b167ea977b23997d6e1c8d35369e2ad2109033b91bcd3f1"} Oct 31 03:55:45.419188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.419146 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:55:45.473105 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.473054 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-maas-api-tls\") pod \"315c9d44-5d61-47a5-8bf7-85172dea8af2\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " Oct 31 03:55:45.473924 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.473899 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-metrics-certs\") pod \"315c9d44-5d61-47a5-8bf7-85172dea8af2\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " Oct 31 03:55:45.474624 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.474604 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315c9d44-5d61-47a5-8bf7-85172dea8af2-kube-api-access-x48jh\" (UniqueName: \"kubernetes.io/projected/315c9d44-5d61-47a5-8bf7-85172dea8af2-kube-api-access-x48jh\") pod \"315c9d44-5d61-47a5-8bf7-85172dea8af2\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " Oct 31 03:55:45.475159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.475128 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/315c9d44-5d61-47a5-8bf7-85172dea8af2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315c9d44-5d61-47a5-8bf7-85172dea8af2-openshift-service-ca\") pod \"315c9d44-5d61-47a5-8bf7-85172dea8af2\" (UID: \"315c9d44-5d61-47a5-8bf7-85172dea8af2\") " Oct 31 03:55:45.475755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.475640 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315c9d44-5d61-47a5-8bf7-85172dea8af2-openshift-service-ca" pod "315c9d44-5d61-47a5-8bf7-85172dea8af2" (UID: "315c9d44-5d61-47a5-8bf7-85172dea8af2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:45.476798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.476775 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315c9d44-5d61-47a5-8bf7-85172dea8af2-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:45.477879 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.477857 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-maas-api-tls" pod "315c9d44-5d61-47a5-8bf7-85172dea8af2" (UID: "315c9d44-5d61-47a5-8bf7-85172dea8af2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:45.478646 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.478627 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315c9d44-5d61-47a5-8bf7-85172dea8af2-kube-api-access-x48jh" pod "315c9d44-5d61-47a5-8bf7-85172dea8af2" (UID: "315c9d44-5d61-47a5-8bf7-85172dea8af2"). InnerVolumeSpecName "kube-api-access-x48jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:45.483913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.483891 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-metrics-certs" pod "315c9d44-5d61-47a5-8bf7-85172dea8af2" (UID: "315c9d44-5d61-47a5-8bf7-85172dea8af2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:45.578842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.578804 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:45.578842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.578840 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315c9d44-5d61-47a5-8bf7-85172dea8af2-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:45.579134 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:45.578857 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48jh\" (UniqueName: \"kubernetes.io/projected/315c9d44-5d61-47a5-8bf7-85172dea8af2-kube-api-access-x48jh\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:46.073945 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:46.073897 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" Oct 31 03:55:46.074633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:46.073890 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh" event={"ID":"315c9d44-5d61-47a5-8bf7-85172dea8af2","Type":"ContainerDied","Data":"c5bebca95dbc7a3a28b671591e488014408f1fbf1f26db38dd0e01eb6ca050ba"} Oct 31 03:55:46.074633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:46.074102 5520 scope.go:122] "RemoveContainer" containerID="10fb61703c5ba4849b167ea977b23997d6e1c8d35369e2ad2109033b91bcd3f1" Oct 31 03:55:46.111297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:46.111256 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh"] Oct 31 03:55:46.120140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:46.120109 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b5f2-74f98475c6-8pgxh"] Oct 31 03:55:46.724451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:46.724388 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" podUID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" containerName="main" containerID="cri-o://3cded62a839653c584eeef05ed4462c3ec15fa3337f8ef9c6529cecda6af37ec" gracePeriod=45 Oct 31 03:55:46.934226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:46.933629 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" podUID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.217:8000/ready\": dial tcp 10.134.0.217:8000: connect: connection refused" Oct 31 03:55:47.165010 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.164915 5520 generic.go:356] "Generic (PLEG): container finished" podID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" containerID="3cded62a839653c584eeef05ed4462c3ec15fa3337f8ef9c6529cecda6af37ec" exitCode=0 Oct 31 03:55:47.165671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.165644 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" event={"ID":"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6","Type":"ContainerDied","Data":"3cded62a839653c584eeef05ed4462c3ec15fa3337f8ef9c6529cecda6af37ec"} Oct 31 03:55:47.482229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.478192 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" containerID="cri-o://05d68fad773c334bdd1f0e9b0ef116c675b1ce9fc7daeb18bb142f6126bc644b" gracePeriod=45 Oct 31 03:55:47.575720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.575680 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" podUID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" containerName="main" containerID="cri-o://2df23654114c354e0e7a28279dd2a9d20f7eb54268bf5c12163c0f7f7bf64486" gracePeriod=45 Oct 31 03:55:47.754550 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.754481 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:55:47.829235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.828796 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" podUID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": dial tcp 10.134.0.216:8000: connect: connection refused" Oct 31 03:55:47.843800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.843761 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" containerID="cri-o://8565c5e340ed8f276acfa769575b6fd17bd97c350e0c68d3b3c7040ad434580d" gracePeriod=45 Oct 31 03:55:47.871786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.871751 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tmp-dir\") pod \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " Oct 31 03:55:47.895299 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.881611 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-dshm\") pod \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " Oct 31 03:55:47.895638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.895604 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-home\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-home\") pod \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " Oct 31 03:55:48.215515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.213668 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-model-cache\") pod \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " Oct 31 03:55:48.215515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.213923 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-kube-api-access-b4r6c\" (UniqueName: \"kubernetes.io/projected/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-kube-api-access-b4r6c\") pod \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " Oct 31 03:55:48.215515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.214033 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tls-certs\") pod \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\" (UID: \"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6\") " Oct 31 03:55:48.215515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.881421 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tmp-dir" pod "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" (UID: "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:48.215515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.902302 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-home" pod "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" (UID: "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:48.215515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:47.955149 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-dshm" pod "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" (UID: "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:48.219469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.219428 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-model-cache" pod "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" (UID: "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:48.250833 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.250794 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.214:8000/ready\": dial tcp 10.134.0.214:8000: connect: connection refused" Oct 31 03:55:48.266493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.257962 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" containerID="cri-o://a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942" gracePeriod=45 Oct 31 03:55:48.398419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.367398 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:48.398419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.367437 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:48.398419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.367450 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:48.398419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.367463 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:48.552751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.552587 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" containerID="cri-o://9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f" gracePeriod=45 Oct 31 03:55:48.619021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.602482 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" Oct 31 03:55:48.619021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.617033 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="315c9d44-5d61-47a5-8bf7-85172dea8af2" path="/var/lib/kubelet/pods/315c9d44-5d61-47a5-8bf7-85172dea8af2/volumes" Oct 31 03:55:48.655996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.653008 5520 generic.go:356] "Generic (PLEG): container finished" podID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" containerID="2df23654114c354e0e7a28279dd2a9d20f7eb54268bf5c12163c0f7f7bf64486" exitCode=0 Oct 31 03:55:48.655996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.655279 5520 generic.go:356] "Generic (PLEG): container finished" podID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerID="05d68fad773c334bdd1f0e9b0ef116c675b1ce9fc7daeb18bb142f6126bc644b" exitCode=0 Oct 31 03:55:48.679401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.679362 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tls-certs" pod "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" (UID: "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:48.680015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.679990 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x"] Oct 31 03:55:48.680231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.680189 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s" event={"ID":"d76db6e7-f346-42cc-a8bf-c423ab6ab4f6","Type":"ContainerDied","Data":"15f84b4866649b5c17ab215c5f8f8f0b749dc6172a48528d3b74b6b61f2122e8"} Oct 31 03:55:48.680387 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.680367 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" event={"ID":"c13fcad4-2c72-4249-b39c-5ae3d819fd20","Type":"ContainerDied","Data":"2df23654114c354e0e7a28279dd2a9d20f7eb54268bf5c12163c0f7f7bf64486"} Oct 31 03:55:48.686397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.686372 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" event={"ID":"276cbeb2-96f4-43d7-ad78-e77a0323f96a","Type":"ContainerDied","Data":"05d68fad773c334bdd1f0e9b0ef116c675b1ce9fc7daeb18bb142f6126bc644b"} Oct 31 03:55:48.686548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.683851 5520 scope.go:122] "RemoveContainer" containerID="3cded62a839653c584eeef05ed4462c3ec15fa3337f8ef9c6529cecda6af37ec" Oct 31 03:55:48.690338 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.681675 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" containerID="cri-o://9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc" gracePeriod=30 Oct 31 03:55:48.705788 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.705730 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-kube-api-access-b4r6c" pod "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" (UID: "d76db6e7-f346-42cc-a8bf-c423ab6ab4f6"). InnerVolumeSpecName "kube-api-access-b4r6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:48.777454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.764403 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4r6c\" (UniqueName: \"kubernetes.io/projected/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-kube-api-access-b4r6c\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:48.777754 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.777726 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:48.797235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.789086 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": dial tcp 10.134.0.213:8000: connect: connection refused" Oct 31 03:55:48.906190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.905714 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:55:48.959453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:48.953410 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:55:49.122227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.110985 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tls-certs\" (UniqueName: \"kubernetes.io/secret/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tls-certs\") pod \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " Oct 31 03:55:49.123046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.123021 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tmp-dir\") pod \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " Oct 31 03:55:49.123259 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.123242 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13fcad4-2c72-4249-b39c-5ae3d819fd20-kube-api-access-rqwgw\" (UniqueName: \"kubernetes.io/projected/c13fcad4-2c72-4249-b39c-5ae3d819fd20-kube-api-access-rqwgw\") pod \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " Oct 31 03:55:49.123604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.123588 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-model-cache\") pod \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " Oct 31 03:55:49.123738 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.123726 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-dshm\") pod \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " Oct 31 03:55:49.137272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.133796 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tls-certs\" (UniqueName: \"kubernetes.io/secret/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tls-certs\") pod \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " Oct 31 03:55:49.137272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.133848 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tmp-dir\") pod \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " Oct 31 03:55:49.137272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.133900 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-dshm\") pod \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " Oct 31 03:55:49.137272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.133981 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276cbeb2-96f4-43d7-ad78-e77a0323f96a-kube-api-access-gh2gb\" (UniqueName: \"kubernetes.io/projected/276cbeb2-96f4-43d7-ad78-e77a0323f96a-kube-api-access-gh2gb\") pod \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " Oct 31 03:55:49.137272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.134082 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-model-cache\") pod \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " Oct 31 03:55:49.137272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.134165 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-home\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-home\") pod \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\" (UID: \"276cbeb2-96f4-43d7-ad78-e77a0323f96a\") " Oct 31 03:55:49.137272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.134235 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-home\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-home\") pod \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\" (UID: \"c13fcad4-2c72-4249-b39c-5ae3d819fd20\") " Oct 31 03:55:49.141415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.141380 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-home" pod "c13fcad4-2c72-4249-b39c-5ae3d819fd20" (UID: "c13fcad4-2c72-4249-b39c-5ae3d819fd20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.164090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.163699 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-model-cache" pod "c13fcad4-2c72-4249-b39c-5ae3d819fd20" (UID: "c13fcad4-2c72-4249-b39c-5ae3d819fd20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.192274 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.191173 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tmp-dir" pod "276cbeb2-96f4-43d7-ad78-e77a0323f96a" (UID: "276cbeb2-96f4-43d7-ad78-e77a0323f96a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.251653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.251602 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-model-cache" pod "276cbeb2-96f4-43d7-ad78-e77a0323f96a" (UID: "276cbeb2-96f4-43d7-ad78-e77a0323f96a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.252665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.252627 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-home" pod "276cbeb2-96f4-43d7-ad78-e77a0323f96a" (UID: "276cbeb2-96f4-43d7-ad78-e77a0323f96a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.253986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.253959 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.281296 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.281829 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tmp-dir" pod "c13fcad4-2c72-4249-b39c-5ae3d819fd20" (UID: "c13fcad4-2c72-4249-b39c-5ae3d819fd20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.314819 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.314853 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.314872 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.314886 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.314900 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.349934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.314914 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.452316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.395640 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.525474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.499827 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.525474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.509564 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": dial tcp 10.134.0.212:8000: connect: connection refused" Oct 31 03:55:49.525474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.509673 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:55:49.543485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.529357 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 03:55:49.543485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.529680 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.560572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.553991 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 03:55:49.575465 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.573588 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13fcad4-2c72-4249-b39c-5ae3d819fd20-kube-api-access-rqwgw" pod "c13fcad4-2c72-4249-b39c-5ae3d819fd20" (UID: "c13fcad4-2c72-4249-b39c-5ae3d819fd20"). InnerVolumeSpecName "kube-api-access-rqwgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:49.605515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.598560 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-dshm" pod "276cbeb2-96f4-43d7-ad78-e77a0323f96a" (UID: "276cbeb2-96f4-43d7-ad78-e77a0323f96a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.605515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.600182 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276cbeb2-96f4-43d7-ad78-e77a0323f96a-kube-api-access-gh2gb" pod "276cbeb2-96f4-43d7-ad78-e77a0323f96a" (UID: "276cbeb2-96f4-43d7-ad78-e77a0323f96a"). InnerVolumeSpecName "kube-api-access-gh2gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:49.624363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.617168 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqwgw\" (UniqueName: \"kubernetes.io/projected/c13fcad4-2c72-4249-b39c-5ae3d819fd20-kube-api-access-rqwgw\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.624363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.617226 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276cbeb2-96f4-43d7-ad78-e77a0323f96a-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.624363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.617244 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh2gb\" (UniqueName: \"kubernetes.io/projected/276cbeb2-96f4-43d7-ad78-e77a0323f96a-kube-api-access-gh2gb\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.690697 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.689472 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s"] Oct 31 03:55:49.715751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.713875 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-63258b/auth-test-model-63258b-kserve-96fb788fc-c7x6s"] Oct 31 03:55:49.715751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.714321 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-dshm" pod "c13fcad4-2c72-4249-b39c-5ae3d819fd20" (UID: "c13fcad4-2c72-4249-b39c-5ae3d819fd20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:49.750455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.716032 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" containerID="cri-o://d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73" gracePeriod=45 Oct 31 03:55:49.750455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.738724 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c13fcad4-2c72-4249-b39c-5ae3d819fd20-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.772240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.757841 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" containerID="cri-o://1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80" gracePeriod=45 Oct 31 03:55:49.772240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.759689 5520 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 31 03:55:49.772240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.761969 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": dial tcp 10.134.0.213:8000: connect: connection refused" Oct 31 03:55:49.772240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.762068 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:55:49.802823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.793080 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tls-certs" pod "276cbeb2-96f4-43d7-ad78-e77a0323f96a" (UID: "276cbeb2-96f4-43d7-ad78-e77a0323f96a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:49.808178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.807977 5520 generic.go:356] "Generic (PLEG): container finished" podID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerID="8565c5e340ed8f276acfa769575b6fd17bd97c350e0c68d3b3c7040ad434580d" exitCode=0 Oct 31 03:55:49.808178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.808127 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" event={"ID":"61ef4a9f-a824-4884-a338-f5e22314eeee","Type":"ContainerDied","Data":"8565c5e340ed8f276acfa769575b6fd17bd97c350e0c68d3b3c7040ad434580d"} Oct 31 03:55:49.833440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.832702 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" event={"ID":"c13fcad4-2c72-4249-b39c-5ae3d819fd20","Type":"ContainerDied","Data":"4458cf654ed24332223137d96736a5c731c3e1cee81110329e5b9bd8163c6e41"} Oct 31 03:55:49.833440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.832765 5520 scope.go:122] "RemoveContainer" containerID="2df23654114c354e0e7a28279dd2a9d20f7eb54268bf5c12163c0f7f7bf64486" Oct 31 03:55:49.833440 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.832869 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:55:49.843000 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.842959 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276cbeb2-96f4-43d7-ad78-e77a0323f96a-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:49.846228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.845819 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" event={"ID":"276cbeb2-96f4-43d7-ad78-e77a0323f96a","Type":"ContainerDied","Data":"f9973ab2904af555b48e082a03dedf81d82bd82eba648f495f75b65430f1e144"} Oct 31 03:55:49.846228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.845914 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp" Oct 31 03:55:49.870529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.870232 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" podUID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.871463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.870884 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf" Oct 31 03:55:49.871463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.871407 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tls-certs" pod "c13fcad4-2c72-4249-b39c-5ae3d819fd20" (UID: "c13fcad4-2c72-4249-b39c-5ae3d819fd20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:49.978839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:49.978776 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13fcad4-2c72-4249-b39c-5ae3d819fd20-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.073172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.072368 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:50.107240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.095173 5520 scope.go:122] "RemoveContainer" containerID="05d68fad773c334bdd1f0e9b0ef116c675b1ce9fc7daeb18bb142f6126bc644b" Oct 31 03:55:50.129397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.129250 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:55:50.130533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.130340 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" path="/var/lib/kubelet/pods/d76db6e7-f346-42cc-a8bf-c423ab6ab4f6/volumes" Oct 31 03:55:50.191655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.191624 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp"] Oct 31 03:55:50.191939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.191921 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-86a40d/test-model-86a40d-kserve-6dbbdb988f-kflsp"] Oct 31 03:55:50.211294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.211249 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" probeResult="failure" output="Get \"https://10.134.0.214:8000/ready\": context deadline exceeded" Oct 31 03:55:50.211653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.211634 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:55:50.240790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.240593 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-model-cache\") pod \"61ef4a9f-a824-4884-a338-f5e22314eeee\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " Oct 31 03:55:50.241114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.241089 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ef4a9f-a824-4884-a338-f5e22314eeee-kube-api-access-jq7m4\" (UniqueName: \"kubernetes.io/projected/61ef4a9f-a824-4884-a338-f5e22314eeee-kube-api-access-jq7m4\") pod \"61ef4a9f-a824-4884-a338-f5e22314eeee\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " Oct 31 03:55:50.241411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.241381 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ef4a9f-a824-4884-a338-f5e22314eeee-tls-certs\" (UniqueName: \"kubernetes.io/secret/61ef4a9f-a824-4884-a338-f5e22314eeee-tls-certs\") pod \"61ef4a9f-a824-4884-a338-f5e22314eeee\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " Oct 31 03:55:50.241615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.241584 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-home\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-home\") pod \"61ef4a9f-a824-4884-a338-f5e22314eeee\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " Oct 31 03:55:50.241757 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.241744 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-dshm\") pod \"61ef4a9f-a824-4884-a338-f5e22314eeee\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " Oct 31 03:55:50.241896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.241883 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-tmp-dir\") pod \"61ef4a9f-a824-4884-a338-f5e22314eeee\" (UID: \"61ef4a9f-a824-4884-a338-f5e22314eeee\") " Oct 31 03:55:50.243120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.243098 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-tmp-dir" pod "61ef4a9f-a824-4884-a338-f5e22314eeee" (UID: "61ef4a9f-a824-4884-a338-f5e22314eeee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.243465 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.243445 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-model-cache" pod "61ef4a9f-a824-4884-a338-f5e22314eeee" (UID: "61ef4a9f-a824-4884-a338-f5e22314eeee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.244660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.244637 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-home" pod "61ef4a9f-a824-4884-a338-f5e22314eeee" (UID: "61ef4a9f-a824-4884-a338-f5e22314eeee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.270483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.267356 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-dshm" pod "61ef4a9f-a824-4884-a338-f5e22314eeee" (UID: "61ef4a9f-a824-4884-a338-f5e22314eeee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.270483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.269708 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ef4a9f-a824-4884-a338-f5e22314eeee-tls-certs" pod "61ef4a9f-a824-4884-a338-f5e22314eeee" (UID: "61ef4a9f-a824-4884-a338-f5e22314eeee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:50.287415 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.286306 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf"] Oct 31 03:55:50.304138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.303876 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-86a834/auth-test-model-86a834-kserve-58d79bdbb9-ql6wf"] Oct 31 03:55:50.320876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.320811 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ef4a9f-a824-4884-a338-f5e22314eeee-kube-api-access-jq7m4" pod "61ef4a9f-a824-4884-a338-f5e22314eeee" (UID: "61ef4a9f-a824-4884-a338-f5e22314eeee"). InnerVolumeSpecName "kube-api-access-jq7m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:50.342002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.341517 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": dial tcp 10.134.0.197:8000: connect: connection refused" Oct 31 03:55:50.344535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.344342 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.344535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.344405 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7m4\" (UniqueName: \"kubernetes.io/projected/61ef4a9f-a824-4884-a338-f5e22314eeee-kube-api-access-jq7m4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.344535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.344422 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ef4a9f-a824-4884-a338-f5e22314eeee-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.344535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.344435 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.344535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.344449 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.345330 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.344465 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ef4a9f-a824-4884-a338-f5e22314eeee-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.455253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.454766 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:55:50.552332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.552245 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tls-certs\") pod \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " Oct 31 03:55:50.552572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.552550 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tmp-dir\") pod \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " Oct 31 03:55:50.552666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.552627 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-kube-api-access-qh69s\" (UniqueName: \"kubernetes.io/projected/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-kube-api-access-qh69s\") pod \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " Oct 31 03:55:50.552728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.552699 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-dshm\") pod \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " Oct 31 03:55:50.552785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.552746 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-model-cache\") pod \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " Oct 31 03:55:50.552958 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.552943 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-home\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-home\") pod \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\" (UID: \"ab4f5aae-be3b-4e3b-b0df-a66c622e950a\") " Oct 31 03:55:50.554479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.554447 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-home" pod "ab4f5aae-be3b-4e3b-b0df-a66c622e950a" (UID: "ab4f5aae-be3b-4e3b-b0df-a66c622e950a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.565321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.557303 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tmp-dir" pod "ab4f5aae-be3b-4e3b-b0df-a66c622e950a" (UID: "ab4f5aae-be3b-4e3b-b0df-a66c622e950a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.565321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.565266 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-model-cache" pod "ab4f5aae-be3b-4e3b-b0df-a66c622e950a" (UID: "ab4f5aae-be3b-4e3b-b0df-a66c622e950a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.588030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.587988 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-kube-api-access-qh69s" pod "ab4f5aae-be3b-4e3b-b0df-a66c622e950a" (UID: "ab4f5aae-be3b-4e3b-b0df-a66c622e950a"). InnerVolumeSpecName "kube-api-access-qh69s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:50.588611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.588569 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-dshm" pod "ab4f5aae-be3b-4e3b-b0df-a66c622e950a" (UID: "ab4f5aae-be3b-4e3b-b0df-a66c622e950a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.588611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.588600 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tls-certs" pod "ab4f5aae-be3b-4e3b-b0df-a66c622e950a" (UID: "ab4f5aae-be3b-4e3b-b0df-a66c622e950a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:50.606837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.606808 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:55:50.609911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.609889 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:55:50.626495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.626468 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:55:50.654977 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.654928 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-model-cache\") pod \"216f6f37-f90a-45c2-836f-a596db878cf1\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " Oct 31 03:55:50.655162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.655067 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216f6f37-f90a-45c2-836f-a596db878cf1-kube-api-access-qmqvn\" (UniqueName: \"kubernetes.io/projected/216f6f37-f90a-45c2-836f-a596db878cf1-kube-api-access-qmqvn\") pod \"216f6f37-f90a-45c2-836f-a596db878cf1\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " Oct 31 03:55:50.655316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.655267 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-model-cache" pod "216f6f37-f90a-45c2-836f-a596db878cf1" (UID: "216f6f37-f90a-45c2-836f-a596db878cf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.655528 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.655171 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-model-cache\") pod \"9e552324-bc28-429a-afa7-f882d6356760\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " Oct 31 03:55:50.655607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.655593 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-home\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-home\") pod \"9e552324-bc28-429a-afa7-f882d6356760\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " Oct 31 03:55:50.655670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.655642 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-dshm\") pod \"216f6f37-f90a-45c2-836f-a596db878cf1\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " Oct 31 03:55:50.655732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.655721 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-tmp-dir\") pod \"9e552324-bc28-429a-afa7-f882d6356760\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " Oct 31 03:55:50.655849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.655815 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216f6f37-f90a-45c2-836f-a596db878cf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/216f6f37-f90a-45c2-836f-a596db878cf1-tls-certs\") pod \"216f6f37-f90a-45c2-836f-a596db878cf1\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " Oct 31 03:55:50.656083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.656048 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-tmp-dir\") pod \"216f6f37-f90a-45c2-836f-a596db878cf1\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " Oct 31 03:55:50.656285 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.656268 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e552324-bc28-429a-afa7-f882d6356760-kube-api-access-nmfzn\" (UniqueName: \"kubernetes.io/projected/9e552324-bc28-429a-afa7-f882d6356760-kube-api-access-nmfzn\") pod \"9e552324-bc28-429a-afa7-f882d6356760\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " Oct 31 03:55:50.656418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.656403 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-dshm\") pod \"9e552324-bc28-429a-afa7-f882d6356760\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " Oct 31 03:55:50.656630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.656614 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-home\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-home\") pod \"216f6f37-f90a-45c2-836f-a596db878cf1\" (UID: \"216f6f37-f90a-45c2-836f-a596db878cf1\") " Oct 31 03:55:50.656874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.656858 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e552324-bc28-429a-afa7-f882d6356760-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e552324-bc28-429a-afa7-f882d6356760-tls-certs\") pod \"9e552324-bc28-429a-afa7-f882d6356760\" (UID: \"9e552324-bc28-429a-afa7-f882d6356760\") " Oct 31 03:55:50.658504 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.658482 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.658649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.658634 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.658752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.658738 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.658848 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.658837 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.658950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.658937 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.659040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.659027 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.659151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.659135 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh69s\" (UniqueName: \"kubernetes.io/projected/ab4f5aae-be3b-4e3b-b0df-a66c622e950a-kube-api-access-qh69s\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.659266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.658496 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f6f37-f90a-45c2-836f-a596db878cf1-kube-api-access-qmqvn" pod "216f6f37-f90a-45c2-836f-a596db878cf1" (UID: "216f6f37-f90a-45c2-836f-a596db878cf1"). InnerVolumeSpecName "kube-api-access-qmqvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:50.659742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.659621 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-tmp-dir" pod "216f6f37-f90a-45c2-836f-a596db878cf1" (UID: "216f6f37-f90a-45c2-836f-a596db878cf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.660089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.659956 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-tmp-dir" pod "9e552324-bc28-429a-afa7-f882d6356760" (UID: "9e552324-bc28-429a-afa7-f882d6356760"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.660268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.660166 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-model-cache" pod "9e552324-bc28-429a-afa7-f882d6356760" (UID: "9e552324-bc28-429a-afa7-f882d6356760"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.660268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.660191 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-home" pod "9e552324-bc28-429a-afa7-f882d6356760" (UID: "9e552324-bc28-429a-afa7-f882d6356760"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.660484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.660458 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-home" pod "216f6f37-f90a-45c2-836f-a596db878cf1" (UID: "216f6f37-f90a-45c2-836f-a596db878cf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.667919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.667878 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-dshm" pod "216f6f37-f90a-45c2-836f-a596db878cf1" (UID: "216f6f37-f90a-45c2-836f-a596db878cf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.668797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.668762 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216f6f37-f90a-45c2-836f-a596db878cf1-tls-certs" pod "216f6f37-f90a-45c2-836f-a596db878cf1" (UID: "216f6f37-f90a-45c2-836f-a596db878cf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:50.668797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.668775 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e552324-bc28-429a-afa7-f882d6356760-tls-certs" pod "9e552324-bc28-429a-afa7-f882d6356760" (UID: "9e552324-bc28-429a-afa7-f882d6356760"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:50.669053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.669029 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-dshm" pod "9e552324-bc28-429a-afa7-f882d6356760" (UID: "9e552324-bc28-429a-afa7-f882d6356760"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.671030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.671007 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e552324-bc28-429a-afa7-f882d6356760-kube-api-access-nmfzn" pod "9e552324-bc28-429a-afa7-f882d6356760" (UID: "9e552324-bc28-429a-afa7-f882d6356760"). InnerVolumeSpecName "kube-api-access-nmfzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:50.760459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760428 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-model-cache\") pod \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " Oct 31 03:55:50.760642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760481 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tmp-dir\") pod \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " Oct 31 03:55:50.760642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760515 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tls-certs\") pod \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " Oct 31 03:55:50.760766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760686 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-kube-api-access-wlmf2\" (UniqueName: \"kubernetes.io/projected/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-kube-api-access-wlmf2\") pod \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " Oct 31 03:55:50.760766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760734 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-dshm\") pod \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " Oct 31 03:55:50.760766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760739 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-model-cache" pod "23eddc40-82f6-4177-92d2-6bbcfcca1c2a" (UID: "23eddc40-82f6-4177-92d2-6bbcfcca1c2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.760766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760749 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tmp-dir" pod "23eddc40-82f6-4177-92d2-6bbcfcca1c2a" (UID: "23eddc40-82f6-4177-92d2-6bbcfcca1c2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.760996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.760863 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-home\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-home\") pod \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\" (UID: \"23eddc40-82f6-4177-92d2-6bbcfcca1c2a\") " Oct 31 03:55:50.761154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761128 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-home" pod "23eddc40-82f6-4177-92d2-6bbcfcca1c2a" (UID: "23eddc40-82f6-4177-92d2-6bbcfcca1c2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.761554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761537 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761557 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761567 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e552324-bc28-429a-afa7-f882d6356760-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761576 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761584 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761593 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmqvn\" (UniqueName: \"kubernetes.io/projected/216f6f37-f90a-45c2-836f-a596db878cf1-kube-api-access-qmqvn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761603 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761615 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761625 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761633 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761640 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216f6f37-f90a-45c2-836f-a596db878cf1-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761648 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216f6f37-f90a-45c2-836f-a596db878cf1-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761657 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmfzn\" (UniqueName: \"kubernetes.io/projected/9e552324-bc28-429a-afa7-f882d6356760-kube-api-access-nmfzn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.761808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.761664 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e552324-bc28-429a-afa7-f882d6356760-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.763547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.763517 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tls-certs" pod "23eddc40-82f6-4177-92d2-6bbcfcca1c2a" (UID: "23eddc40-82f6-4177-92d2-6bbcfcca1c2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:50.763637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.763569 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-dshm" pod "23eddc40-82f6-4177-92d2-6bbcfcca1c2a" (UID: "23eddc40-82f6-4177-92d2-6bbcfcca1c2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:50.763637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.763599 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-kube-api-access-wlmf2" pod "23eddc40-82f6-4177-92d2-6bbcfcca1c2a" (UID: "23eddc40-82f6-4177-92d2-6bbcfcca1c2a"). InnerVolumeSpecName "kube-api-access-wlmf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:50.855019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.854939 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" Oct 31 03:55:50.855019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.854960 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz" event={"ID":"61ef4a9f-a824-4884-a338-f5e22314eeee","Type":"ContainerDied","Data":"3444a062ddf8c5c15f7b8a3c124549420b21438cedd7d7419cfb1d0b5de25330"} Oct 31 03:55:50.855019 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.855011 5520 scope.go:122] "RemoveContainer" containerID="8565c5e340ed8f276acfa769575b6fd17bd97c350e0c68d3b3c7040ad434580d" Oct 31 03:55:50.856603 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.856582 5520 generic.go:356] "Generic (PLEG): container finished" podID="216f6f37-f90a-45c2-836f-a596db878cf1" containerID="9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f" exitCode=0 Oct 31 03:55:50.856748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.856650 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" Oct 31 03:55:50.856748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.856667 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" event={"ID":"216f6f37-f90a-45c2-836f-a596db878cf1","Type":"ContainerDied","Data":"9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f"} Oct 31 03:55:50.856748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.856692 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254" event={"ID":"216f6f37-f90a-45c2-836f-a596db878cf1","Type":"ContainerDied","Data":"996a2d2c40745363a7b0e5f56bf80e55b8b3766a13c46c1f62893ac543b66d15"} Oct 31 03:55:50.861441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.861413 5520 generic.go:356] "Generic (PLEG): container finished" podID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerID="a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942" exitCode=0 Oct 31 03:55:50.861574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.861513 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" event={"ID":"ab4f5aae-be3b-4e3b-b0df-a66c622e950a","Type":"ContainerDied","Data":"a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942"} Oct 31 03:55:50.861574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.861541 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" event={"ID":"ab4f5aae-be3b-4e3b-b0df-a66c622e950a","Type":"ContainerDied","Data":"4a5ac083e50cc3f906671d2b66f5e50b64cd8d043d79fa621f0e275d36f4671b"} Oct 31 03:55:50.861574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.861540 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" Oct 31 03:55:50.862919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.862899 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmf2\" (UniqueName: \"kubernetes.io/projected/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-kube-api-access-wlmf2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.863026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.862923 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.863026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.862938 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23eddc40-82f6-4177-92d2-6bbcfcca1c2a-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:50.871243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.865276 5520 generic.go:356] "Generic (PLEG): container finished" podID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerID="1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80" exitCode=0 Oct 31 03:55:50.871243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.865378 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" event={"ID":"23eddc40-82f6-4177-92d2-6bbcfcca1c2a","Type":"ContainerDied","Data":"1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80"} Oct 31 03:55:50.871243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.865407 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" event={"ID":"23eddc40-82f6-4177-92d2-6bbcfcca1c2a","Type":"ContainerDied","Data":"9c5224b1c0d71a4cfd278cb4e15e866c0baabdf01e35051238097c1d89493692"} Oct 31 03:55:50.871243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.865380 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6" Oct 31 03:55:50.871243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.867334 5520 scope.go:122] "RemoveContainer" containerID="9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f" Oct 31 03:55:50.872043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.871900 5520 generic.go:356] "Generic (PLEG): container finished" podID="9e552324-bc28-429a-afa7-f882d6356760" containerID="d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73" exitCode=0 Oct 31 03:55:50.872043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.871973 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" event={"ID":"9e552324-bc28-429a-afa7-f882d6356760","Type":"ContainerDied","Data":"d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73"} Oct 31 03:55:50.872043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.871997 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" event={"ID":"9e552324-bc28-429a-afa7-f882d6356760","Type":"ContainerDied","Data":"2f89b709a863f8fc68dbb3b6ef2f176770895ffa504617f5059c408c0817eed7"} Oct 31 03:55:50.872335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.872050 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx" Oct 31 03:55:50.886613 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.886425 5520 scope.go:122] "RemoveContainer" containerID="9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f" Oct 31 03:55:50.886748 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:50.886718 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f\": container with ID starting with 9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f not found: ID does not exist" containerID="9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f" Oct 31 03:55:50.886875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.886753 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f"} err="failed to get container status \"9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f\": rpc error: code = NotFound desc = could not find container \"9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f\": container with ID starting with 9d6044946089850c4459656bcb1eee9388f4b659c094bd87efdd2079863b3f9f not found: ID does not exist" Oct 31 03:55:50.886875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.886816 5520 scope.go:122] "RemoveContainer" containerID="a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942" Oct 31 03:55:50.899280 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.899255 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz"] Oct 31 03:55:50.900228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.900188 5520 scope.go:122] "RemoveContainer" containerID="a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942" Oct 31 03:55:50.900579 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:50.900552 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942\": container with ID starting with a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942 not found: ID does not exist" containerID="a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942" Oct 31 03:55:50.900680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.900588 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942"} err="failed to get container status \"a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942\": rpc error: code = NotFound desc = could not find container \"a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942\": container with ID starting with a8c4cb3b9690281d9c5b46ab75e12b4a07ce4b1858e6ce9271b5fdb5a0112942 not found: ID does not exist" Oct 31 03:55:50.900680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.900610 5520 scope.go:122] "RemoveContainer" containerID="1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80" Oct 31 03:55:50.901691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.901419 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ce9b89/test-model-ce9b89-kserve-8fbfd9b9f-kxpzz"] Oct 31 03:55:50.906081 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.906061 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254"] Oct 31 03:55:50.909332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.909303 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b81d65/test-model-b81d65-kserve-76f84945d7-hg254"] Oct 31 03:55:50.914478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.914430 5520 scope.go:122] "RemoveContainer" containerID="1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80" Oct 31 03:55:50.914786 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:50.914755 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80\": container with ID starting with 1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80 not found: ID does not exist" containerID="1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80" Oct 31 03:55:50.914899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.914795 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80"} err="failed to get container status \"1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80\": rpc error: code = NotFound desc = could not find container \"1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80\": container with ID starting with 1cd14c37e77c023d616558ad5d081fb82ee128a1aef7f1fe8c1374ff42515d80 not found: ID does not exist" Oct 31 03:55:50.914899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.914819 5520 scope.go:122] "RemoveContainer" containerID="d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73" Oct 31 03:55:50.920859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.920839 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6"] Oct 31 03:55:50.924192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.924172 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a5d9b6/test-model-a5d9b6-kserve-79b884f49-v4lq6"] Oct 31 03:55:50.927911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.927894 5520 scope.go:122] "RemoveContainer" containerID="d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73" Oct 31 03:55:50.928268 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:50.928244 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73\": container with ID starting with d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73 not found: ID does not exist" containerID="d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73" Oct 31 03:55:50.928342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.928276 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73"} err="failed to get container status \"d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73\": rpc error: code = NotFound desc = could not find container \"d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73\": container with ID starting with d51d3de5abcd79ddcd2496112b41c7fb87d7fd05a9b1726f6ddf96e64e833c73 not found: ID does not exist" Oct 31 03:55:50.931806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.931786 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2"] Oct 31 03:55:50.934727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.934708 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2"] Oct 31 03:55:50.945759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.945658 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx"] Oct 31 03:55:50.951676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:50.951651 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b68780/test-model-b68780-kserve-55b76b45cc-569bx"] Oct 31 03:55:51.443904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.443827 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-afb8be/auth-test-model-afb8be-kserve-5fd464f665-p24m2" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": context deadline exceeded" Oct 31 03:55:51.923131 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.923048 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" path="/var/lib/kubelet/pods/216f6f37-f90a-45c2-836f-a596db878cf1/volumes" Oct 31 03:55:51.923673 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.923654 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" path="/var/lib/kubelet/pods/23eddc40-82f6-4177-92d2-6bbcfcca1c2a/volumes" Oct 31 03:55:51.924127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.924112 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" path="/var/lib/kubelet/pods/276cbeb2-96f4-43d7-ad78-e77a0323f96a/volumes" Oct 31 03:55:51.924876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.924860 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" path="/var/lib/kubelet/pods/61ef4a9f-a824-4884-a338-f5e22314eeee/volumes" Oct 31 03:55:51.925293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.925279 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e552324-bc28-429a-afa7-f882d6356760" path="/var/lib/kubelet/pods/9e552324-bc28-429a-afa7-f882d6356760/volumes" Oct 31 03:55:51.925676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.925663 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" path="/var/lib/kubelet/pods/ab4f5aae-be3b-4e3b-b0df-a66c622e950a/volumes" Oct 31 03:55:51.926462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:51.926444 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" path="/var/lib/kubelet/pods/c13fcad4-2c72-4249-b39c-5ae3d819fd20/volumes" Oct 31 03:55:54.682080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.682053 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:55:54.827809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.827769 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-socket\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.827911 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-podinfo\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.827979 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-kube-api-access-q2hlh\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-kube-api-access-q2hlh\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828024 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-certs\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828029 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-socket" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:54.828149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828052 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istiod-ca-cert\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828092 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-credential-socket\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828120 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-data\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828267 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-token\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-token\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.828432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828406 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-certs" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:54.828549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828424 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-credential-socket" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:54.828549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828482 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istiod-ca-cert" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:54.828549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828467 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-data" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:54.828549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.828520 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-envoy\") pod \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\" (UID: \"fcb74b35-d0cc-4865-9617-263b41b4ec7a\") " Oct 31 03:55:54.829557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.829530 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.829691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.829562 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.829691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.829578 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.829691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.829595 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.829691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.829626 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.830431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.830403 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-kube-api-access-q2hlh" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "kube-api-access-q2hlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:54.830567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.830491 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-podinfo" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:55:54.830644 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.830628 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-token" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:54.830803 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.830783 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-envoy" pod "fcb74b35-d0cc-4865-9617-263b41b4ec7a" (UID: "fcb74b35-d0cc-4865-9617-263b41b4ec7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:54.909464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.909429 5520 generic.go:356] "Generic (PLEG): container finished" podID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerID="9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc" exitCode=0 Oct 31 03:55:54.909687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.909489 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" Oct 31 03:55:54.909687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.909524 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" event={"ID":"fcb74b35-d0cc-4865-9617-263b41b4ec7a","Type":"ContainerDied","Data":"9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc"} Oct 31 03:55:54.909687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.909575 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x" event={"ID":"fcb74b35-d0cc-4865-9617-263b41b4ec7a","Type":"ContainerDied","Data":"48fe2fa30016c545b68079ec6748dacbee70f659b2757a5a2bc8a8773065b852"} Oct 31 03:55:54.909687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.909596 5520 scope.go:122] "RemoveContainer" containerID="9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc" Oct 31 03:55:54.922083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.922061 5520 scope.go:122] "RemoveContainer" containerID="75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8" Oct 31 03:55:54.931086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.931020 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2hlh\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-kube-api-access-q2hlh\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.931086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.931048 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.931086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.931058 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.931086 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.931067 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb74b35-d0cc-4865-9617-263b41b4ec7a-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:54.934087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.933988 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x"] Oct 31 03:55:54.935170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.935148 5520 scope.go:122] "RemoveContainer" containerID="9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc" Oct 31 03:55:54.935549 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:54.935529 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc\": container with ID starting with 9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc not found: ID does not exist" containerID="9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc" Oct 31 03:55:54.935640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.935558 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc"} err="failed to get container status \"9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc\": rpc error: code = NotFound desc = could not find container \"9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc\": container with ID starting with 9b55bf0e1ef942b3df3fcc090034e95e78ebc9e1ddbbea1c56b775a0333f1dfc not found: ID does not exist" Oct 31 03:55:54.935640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.935580 5520 scope.go:122] "RemoveContainer" containerID="75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8" Oct 31 03:55:54.935830 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:55:54.935814 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8\": container with ID starting with 75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8 not found: ID does not exist" containerID="75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8" Oct 31 03:55:54.935890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.935834 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8"} err="failed to get container status \"75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8\": rpc error: code = NotFound desc = could not find container \"75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8\": container with ID starting with 75fed34e4f14798e9bbd7f5134da9f615102e28e2d6b1057bd0adcda71614eb8 not found: ID does not exist" Oct 31 03:55:54.940264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:54.940243 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b8fd22-openshift-default-68ffd75b5f-fjs7x"] Oct 31 03:55:55.920031 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:55.919989 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" path="/var/lib/kubelet/pods/fcb74b35-d0cc-4865-9617-263b41b4ec7a/volumes" Oct 31 03:55:58.037772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.037731 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx"] Oct 31 03:55:58.038810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.038756 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" containerID="cri-o://4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73" gracePeriod=30 Oct 31 03:55:58.852050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852010 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk"] Oct 31 03:55:58.852900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852873 5520 cpu_manager.go:414] "containerMap: removing container" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" Oct 31 03:55:58.852900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852897 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852926 5520 cpu_manager.go:414] "containerMap: removing container" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852934 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852947 5520 cpu_manager.go:414] "containerMap: removing container" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852953 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852965 5520 cpu_manager.go:414] "containerMap: removing container" podUID="315c9d44-5d61-47a5-8bf7-85172dea8af2" containerName="maas-api" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852972 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315c9d44-5d61-47a5-8bf7-85172dea8af2" containerName="maas-api" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852981 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bd63b361-66b8-4cc5-a8db-619eaf42496b" containerName="payload-pre-processing" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.852988 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd63b361-66b8-4cc5-a8db-619eaf42496b" containerName="payload-pre-processing" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853001 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853008 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853022 5520 cpu_manager.go:414] "containerMap: removing container" podUID="52dbfac7-97ba-468a-b635-ba2a7faf6ef1" containerName="payload-processing" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853028 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52dbfac7-97ba-468a-b635-ba2a7faf6ef1" containerName="payload-processing" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853043 5520 cpu_manager.go:414] "containerMap: removing container" podUID="a4b73f30-4c6f-4f53-a679-84694df92392" containerName="payload-processing" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853050 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b73f30-4c6f-4f53-a679-84694df92392" containerName="payload-processing" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853060 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853067 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853080 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853088 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853107 5520 cpu_manager.go:414] "containerMap: removing container" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853113 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" Oct 31 03:55:58.853115 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853124 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e86477ac-83fb-4266-a83f-560f10bdb64c" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853130 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86477ac-83fb-4266-a83f-560f10bdb64c" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853146 5520 cpu_manager.go:414] "containerMap: removing container" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853154 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853166 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853173 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853186 5520 cpu_manager.go:414] "containerMap: removing container" podUID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853192 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853221 5520 cpu_manager.go:414] "containerMap: removing container" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853228 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853260 5520 cpu_manager.go:414] "containerMap: removing container" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853267 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853281 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853288 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853299 5520 cpu_manager.go:414] "containerMap: removing container" podUID="24611e04-1978-4f47-b8d3-4234b0ce003e" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853306 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24611e04-1978-4f47-b8d3-4234b0ce003e" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853322 5520 cpu_manager.go:414] "containerMap: removing container" podUID="50b13be8-fc9d-4cac-9193-658b2bb30eae" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853329 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b13be8-fc9d-4cac-9193-658b2bb30eae" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853340 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853348 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853358 5520 cpu_manager.go:414] "containerMap: removing container" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853364 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853384 5520 cpu_manager.go:414] "containerMap: removing container" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853390 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853403 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853409 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853425 5520 cpu_manager.go:414] "containerMap: removing container" podUID="70694229-8fa5-4e3f-af7f-f875faba310b" containerName="maas-api" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853433 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70694229-8fa5-4e3f-af7f-f875faba310b" containerName="maas-api" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853448 5520 cpu_manager.go:414] "containerMap: removing container" podUID="dbde46c5-a7f8-4d6e-9452-ccda66a974e4" containerName="maas-api" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853456 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbde46c5-a7f8-4d6e-9452-ccda66a974e4" containerName="maas-api" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853619 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="276cbeb2-96f4-43d7-ad78-e77a0323f96a" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853636 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4f5aae-be3b-4e3b-b0df-a66c622e950a" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853645 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="24611e04-1978-4f47-b8d3-4234b0ce003e" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853657 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b13be8-fc9d-4cac-9193-658b2bb30eae" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853667 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853678 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0b94dd-ab27-432b-aa69-5f551939d376" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853687 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853695 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d58c9a-ac48-41b7-bf68-c54fbea4531e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853703 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0d8f04-2f35-4581-a012-9e205b8ced0e" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853713 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="70694229-8fa5-4e3f-af7f-f875faba310b" containerName="maas-api" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853724 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbde46c5-a7f8-4d6e-9452-ccda66a974e4" containerName="maas-api" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853733 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="23eddc40-82f6-4177-92d2-6bbcfcca1c2a" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853745 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853756 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="315c9d44-5d61-47a5-8bf7-85172dea8af2" containerName="maas-api" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853765 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd63b361-66b8-4cc5-a8db-619eaf42496b" containerName="payload-pre-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853775 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="216f6f37-f90a-45c2-836f-a596db878cf1" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853784 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e3fdd4-9152-4bb8-b98d-2445eb671f4b" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853791 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="52dbfac7-97ba-468a-b635-ba2a7faf6ef1" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853802 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b73f30-4c6f-4f53-a679-84694df92392" containerName="payload-processing" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853810 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69df727-93b9-498f-8ff0-07845d18ceea" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853819 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec9a673-d7aa-4b7d-807e-d519c2a29e2c" containerName="istio-proxy" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853830 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e552324-bc28-429a-afa7-f882d6356760" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853838 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86477ac-83fb-4266-a83f-560f10bdb64c" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853848 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ef4a9f-a824-4884-a338-f5e22314eeee" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853857 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13fcad4-2c72-4249-b39c-5ae3d819fd20" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853867 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76db6e7-f346-42cc-a8bf-c423ab6ab4f6" containerName="main" Oct 31 03:55:58.854218 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.853877 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="49484854-701e-49f5-9aa0-85e2338de33c" containerName="istio-proxy" Oct 31 03:55:58.859819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.859791 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:58.870505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.870433 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk"] Oct 31 03:55:58.956046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:58.956015 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-20585c-openshift-default-dockercfg-mnxkq\"" Oct 31 03:55:59.020561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.020512 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.020745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.020611 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.020745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.020686 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.020859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.020776 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.020908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.020887 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.020961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.020936 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.021015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.020998 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8km\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-kube-api-access-gx8km\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.021409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.021389 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.021518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.021493 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123038 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123156 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123298 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123336 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123396 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123433 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123495 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123504 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123524 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123593 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8km\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-kube-api-access-gx8km\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.123680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.123636 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.124293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.124109 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.124293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.124178 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.124293 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.124257 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.125922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.125898 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.126057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.126018 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.131271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.131250 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.131893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.131865 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8km\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-kube-api-access-gx8km\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.184196 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.184156 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:55:59.476662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.476629 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk"] Oct 31 03:55:59.476927 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:55:59.476903 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9ae6cd_e13b_4388_bcf0_2839ffc8bb03.slice/crio-85b104dca20c73bddf8dc6fd801934c0584c7fd19542d947d19dc6f4c8c1667d WatchSource:0}: Error finding container 85b104dca20c73bddf8dc6fd801934c0584c7fd19542d947d19dc6f4c8c1667d: Status 404 returned error can't find the container with id 85b104dca20c73bddf8dc6fd801934c0584c7fd19542d947d19dc6f4c8c1667d Oct 31 03:55:59.480384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.480351 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:55:59.480480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.480416 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:55:59.480480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.480444 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:55:59.962352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.962314 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" event={"ID":"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03","Type":"ContainerStarted","Data":"e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03"} Oct 31 03:55:59.962548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:55:59.962372 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" event={"ID":"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03","Type":"ContainerStarted","Data":"85b104dca20c73bddf8dc6fd801934c0584c7fd19542d947d19dc6f4c8c1667d"} Oct 31 03:56:00.187228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:00.186601 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:56:00.218308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:00.217900 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 31 03:56:00.218308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:00.217958 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 31 03:56:01.200234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:01.197662 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 31 03:56:01.200234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:01.197730 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 31 03:56:02.187228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:02.185315 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 31 03:56:02.187228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:02.185379 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 31 03:56:03.192776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.192727 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 31 03:56:03.193333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.192793 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 31 03:56:03.417465 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.417433 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:56:03.443935 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.443874 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podStartSLOduration=5.443855821 podStartE2EDuration="5.443855821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:00.017724923 +0000 UTC m=+2384.212248044" watchObservedRunningTime="2026-10-31 03:56:03.443855821 +0000 UTC m=+2387.638378944" Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.549327 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-data\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.549432 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-envoy\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.549479 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-token\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-token\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.549616 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istiod-ca-cert\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.549655 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-kube-api-access-4dbtx\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-kube-api-access-4dbtx\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.549738 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-certs\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.549797 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-podinfo\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.550279 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-socket\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.551640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.550440 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-credential-socket\") pod \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\" (UID: \"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf\") " Oct 31 03:56:03.555423 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.555277 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-certs" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:03.555642 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.555611 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-data" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:03.556048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.556021 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istiod-ca-cert" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:03.556409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.556367 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-socket" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:03.556681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.556633 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-credential-socket" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:03.562242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.559696 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.562242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.559728 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.562242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.559744 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.562242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.559759 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.562242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.559776 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.562242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.561980 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-podinfo" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:03.619278 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.609411 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-kube-api-access-4dbtx" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "kube-api-access-4dbtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:03.619449 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.619371 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-envoy" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:03.621336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.620846 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-token" pod "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" (UID: "e5efbc60-6ff8-43eb-bfa4-62687f95d1bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:03.660752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.660717 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.660752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.660748 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.660968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.660759 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dbtx\" (UniqueName: \"kubernetes.io/projected/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-kube-api-access-4dbtx\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:03.660968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:03.660774 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:04.032998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.032965 5520 generic.go:356] "Generic (PLEG): container finished" podID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerID="4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73" exitCode=0 Oct 31 03:56:04.033183 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.033151 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" event={"ID":"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf","Type":"ContainerDied","Data":"4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73"} Oct 31 03:56:04.033246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.033191 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" event={"ID":"e5efbc60-6ff8-43eb-bfa4-62687f95d1bf","Type":"ContainerDied","Data":"95e3c5801597f6b8080bbb925e028383001873418f17cdff38be9c1eebe7939b"} Oct 31 03:56:04.033246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.033238 5520 scope.go:122] "RemoveContainer" containerID="4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73" Oct 31 03:56:04.033374 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.033363 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx" Oct 31 03:56:04.053362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.052660 5520 scope.go:122] "RemoveContainer" containerID="30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c" Oct 31 03:56:04.059137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.059097 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx"] Oct 31 03:56:04.063582 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.063552 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50b5f2-openshift-default-58c68b64dc-7j5kx"] Oct 31 03:56:04.107887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.107855 5520 scope.go:122] "RemoveContainer" containerID="4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73" Oct 31 03:56:04.108359 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:04.108333 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73\": container with ID starting with 4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73 not found: ID does not exist" containerID="4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73" Oct 31 03:56:04.108439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.108371 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73"} err="failed to get container status \"4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73\": rpc error: code = NotFound desc = could not find container \"4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73\": container with ID starting with 4114341db37dfc327aeb3626bcad1c71500a152e80607f3aeda0a02f7d871d73 not found: ID does not exist" Oct 31 03:56:04.108439 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.108397 5520 scope.go:122] "RemoveContainer" containerID="30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c" Oct 31 03:56:04.108770 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:04.108747 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c\": container with ID starting with 30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c not found: ID does not exist" containerID="30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c" Oct 31 03:56:04.108846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.108776 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c"} err="failed to get container status \"30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c\": rpc error: code = NotFound desc = could not find container \"30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c\": container with ID starting with 30466eb99a4c19bfea36ba69ca47dda08adb3b223871e21ffab0d9fa890b040c not found: ID does not exist" Oct 31 03:56:04.198231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.185551 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 31 03:56:04.198231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:04.185609 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 31 03:56:05.187228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:05.186625 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 31 03:56:05.187228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:05.186726 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 31 03:56:06.007800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:06.007765 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" path="/var/lib/kubelet/pods/e5efbc60-6ff8-43eb-bfa4-62687f95d1bf/volumes" Oct 31 03:56:06.185557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:06.185527 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 31 03:56:06.185823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:06.185799 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 31 03:56:07.194150 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:07.194116 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:56:07.194678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:07.194630 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:56:07.195429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:07.195409 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:56:08.757249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:08.754596 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr"] Oct 31 03:56:09.389959 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:09.389919 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:56:09.697257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:09.697171 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn"] Oct 31 03:56:09.697463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:09.697430 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" podUID="5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" containerName="maas-api" containerID="cri-o://24238d1c7c42861303fb9f18163fc82713be300bf934e11d082921522b0a79fe" gracePeriod=30 Oct 31 03:56:10.223224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.222684 5520 generic.go:356] "Generic (PLEG): container finished" podID="5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" containerID="24238d1c7c42861303fb9f18163fc82713be300bf934e11d082921522b0a79fe" exitCode=0 Oct 31 03:56:10.223224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.222829 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" event={"ID":"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54","Type":"ContainerDied","Data":"24238d1c7c42861303fb9f18163fc82713be300bf934e11d082921522b0a79fe"} Oct 31 03:56:10.238237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.233545 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:56:10.287224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.278609 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-maas-api-tls\") pod \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " Oct 31 03:56:10.287224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.279087 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-kube-api-access-msfkj\" (UniqueName: \"kubernetes.io/projected/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-kube-api-access-msfkj\") pod \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " Oct 31 03:56:10.287224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.279131 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-metrics-certs\") pod \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " Oct 31 03:56:10.287224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.279189 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-openshift-service-ca\") pod \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\" (UID: \"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54\") " Oct 31 03:56:10.287224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.281180 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-openshift-service-ca" pod "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" (UID: "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:10.299228 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.293614 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-metrics-certs" pod "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" (UID: "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:10.300537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.300476 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-kube-api-access-msfkj" pod "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" (UID: "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54"). InnerVolumeSpecName "kube-api-access-msfkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:10.300808 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.300773 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-maas-api-tls" pod "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" (UID: "5ea1f583-9a3f-47ae-a2bf-23847dfa9a54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:10.388497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.386458 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:10.388968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.388939 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msfkj\" (UniqueName: \"kubernetes.io/projected/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-kube-api-access-msfkj\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:10.389185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.389167 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:10.389329 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:10.389313 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:11.230923 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:11.230861 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" Oct 31 03:56:11.231455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:11.230888 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn" event={"ID":"5ea1f583-9a3f-47ae-a2bf-23847dfa9a54","Type":"ContainerDied","Data":"958b3bc3cf9156f1c8be92297dd1aa02b050210e37485e3964a8b13b34130b14"} Oct 31 03:56:11.231455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:11.231024 5520 scope.go:122] "RemoveContainer" containerID="24238d1c7c42861303fb9f18163fc82713be300bf934e11d082921522b0a79fe" Oct 31 03:56:11.253772 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:11.253741 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn"] Oct 31 03:56:11.257412 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:11.257389 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b68780-bf8d4d6fd-rqxdn"] Oct 31 03:56:11.948056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:11.948023 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" path="/var/lib/kubelet/pods/5ea1f583-9a3f-47ae-a2bf-23847dfa9a54/volumes" Oct 31 03:56:13.766311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:13.766263 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" podUID="7ed6ab6d-8790-4555-961c-9a00d02bc19a" containerName="payload-processing" containerID="cri-o://6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37" gracePeriod=25 Oct 31 03:56:14.170730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.170701 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:56:14.275350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.275311 5520 generic.go:356] "Generic (PLEG): container finished" podID="7ed6ab6d-8790-4555-961c-9a00d02bc19a" containerID="6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37" exitCode=0 Oct 31 03:56:14.275896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.275358 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" Oct 31 03:56:14.275896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.275398 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" event={"ID":"7ed6ab6d-8790-4555-961c-9a00d02bc19a","Type":"ContainerDied","Data":"6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37"} Oct 31 03:56:14.275896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.275484 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr" event={"ID":"7ed6ab6d-8790-4555-961c-9a00d02bc19a","Type":"ContainerDied","Data":"ba95003c8571c7e8abbb7b487305b9806cec792569e4bcd5a4137a1acfc720f7"} Oct 31 03:56:14.275896 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.275512 5520 scope.go:122] "RemoveContainer" containerID="6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37" Oct 31 03:56:14.286462 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.286444 5520 scope.go:122] "RemoveContainer" containerID="6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37" Oct 31 03:56:14.286755 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:14.286734 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37\": container with ID starting with 6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37 not found: ID does not exist" containerID="6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37" Oct 31 03:56:14.286839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.286762 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37"} err="failed to get container status \"6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37\": rpc error: code = NotFound desc = could not find container \"6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37\": container with ID starting with 6caac90468308b695305126f871b34a16893715b761aae1d7d90548980d43f37 not found: ID does not exist" Oct 31 03:56:14.292193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.292169 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed6ab6d-8790-4555-961c-9a00d02bc19a-kube-api-access-rxgbw\" (UniqueName: \"kubernetes.io/projected/7ed6ab6d-8790-4555-961c-9a00d02bc19a-kube-api-access-rxgbw\") pod \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " Oct 31 03:56:14.292447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.292432 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed6ab6d-8790-4555-961c-9a00d02bc19a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6ab6d-8790-4555-961c-9a00d02bc19a-plugins-config-volume\") pod \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\" (UID: \"7ed6ab6d-8790-4555-961c-9a00d02bc19a\") " Oct 31 03:56:14.292823 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.292804 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed6ab6d-8790-4555-961c-9a00d02bc19a-plugins-config-volume" pod "7ed6ab6d-8790-4555-961c-9a00d02bc19a" (UID: "7ed6ab6d-8790-4555-961c-9a00d02bc19a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:14.293004 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.292989 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6ab6d-8790-4555-961c-9a00d02bc19a-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:14.294429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.294408 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed6ab6d-8790-4555-961c-9a00d02bc19a-kube-api-access-rxgbw" pod "7ed6ab6d-8790-4555-961c-9a00d02bc19a" (UID: "7ed6ab6d-8790-4555-961c-9a00d02bc19a"). InnerVolumeSpecName "kube-api-access-rxgbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:14.394962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.394872 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgbw\" (UniqueName: \"kubernetes.io/projected/7ed6ab6d-8790-4555-961c-9a00d02bc19a-kube-api-access-rxgbw\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:14.604578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.604542 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr"] Oct 31 03:56:14.613716 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:14.613685 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b68780-947cdbcd5-l25lr"] Oct 31 03:56:16.006656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:16.006617 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed6ab6d-8790-4555-961c-9a00d02bc19a" path="/var/lib/kubelet/pods/7ed6ab6d-8790-4555-961c-9a00d02bc19a/volumes" Oct 31 03:56:18.322250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:18.319932 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts"] Oct 31 03:56:18.322250 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:18.320163 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" containerID="cri-o://07ae77079bb80b262c7d4f201d0c4149124c933ce576629c0e5fab757c40174a" gracePeriod=30 Oct 31 03:56:18.383407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:18.376734 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj"] Oct 31 03:56:18.383407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:18.377063 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" containerID="cri-o://fd5512af9ddd7d0e190a73e8d52c1ea3876ada5d8e1aed7a6e72c3e76cc69f15" gracePeriod=30 Oct 31 03:56:19.015749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:19.015707 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98"] Oct 31 03:56:19.376370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:19.376279 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c"] Oct 31 03:56:19.378310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:19.378260 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" containerID="cri-o://0c893f22fa4710f8b4c26fbf783b6c2a992702086104c30e2e30fdd64d28a0f0" gracePeriod=30 Oct 31 03:56:21.520227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:21.512299 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql"] Oct 31 03:56:21.520227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:21.512558 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" containerID="cri-o://cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821" gracePeriod=30 Oct 31 03:56:22.355227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.351351 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr"] Oct 31 03:56:22.355227 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.351699 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" containerID="cri-o://6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a" gracePeriod=30 Oct 31 03:56:22.446233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.427649 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2"] Oct 31 03:56:22.446233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.427942 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" containerID="cri-o://e501340198ca63d97c2d278945a48721672830474f8b248e63919c70fcfc89dc" gracePeriod=30 Oct 31 03:56:22.496012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.495972 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs"] Oct 31 03:56:22.496346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.496316 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" containerID="cri-o://e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c" gracePeriod=30 Oct 31 03:56:22.532263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.531908 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx"] Oct 31 03:56:22.580533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:22.580496 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf"] Oct 31 03:56:23.587986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.587665 5520 generic.go:356] "Generic (PLEG): container finished" podID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerID="fd5512af9ddd7d0e190a73e8d52c1ea3876ada5d8e1aed7a6e72c3e76cc69f15" exitCode=0 Oct 31 03:56:23.587986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.587793 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" event={"ID":"5b9895b9-a592-4acf-a80a-3ef016179bc2","Type":"ContainerDied","Data":"fd5512af9ddd7d0e190a73e8d52c1ea3876ada5d8e1aed7a6e72c3e76cc69f15"} Oct 31 03:56:23.592158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.592118 5520 generic.go:356] "Generic (PLEG): container finished" podID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerID="07ae77079bb80b262c7d4f201d0c4149124c933ce576629c0e5fab757c40174a" exitCode=0 Oct 31 03:56:23.592521 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.592333 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" event={"ID":"1acc8610-820f-4fc1-9fc2-760a4db6a08a","Type":"ContainerDied","Data":"07ae77079bb80b262c7d4f201d0c4149124c933ce576629c0e5fab757c40174a"} Oct 31 03:56:23.698174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.698141 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:56:23.755015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.754987 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:56:23.803936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.803901 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-certs\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804124 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804017 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-podinfo\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804133 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-kube-api-access-g7n5s\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-kube-api-access-g7n5s\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804271 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-certs" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.804336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804323 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-credential-socket\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804400 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-envoy\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804463 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-socket\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804544 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-credential-socket" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.804622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804575 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-token\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-token\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804621 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istiod-ca-cert\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.804749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804699 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-data\") pod \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\" (UID: \"1acc8610-820f-4fc1-9fc2-760a4db6a08a\") " Oct 31 03:56:23.805022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.804985 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-socket" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.805141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.805025 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-data" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.805488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.805464 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.805747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.805494 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.805747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.805508 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.805747 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.805522 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.807348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.807321 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-envoy" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.807348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.807326 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-token" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.807599 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.807582 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-kube-api-access-g7n5s" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "kube-api-access-g7n5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.809859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.809839 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-podinfo" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:23.811327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.811311 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istiod-ca-cert" pod "1acc8610-820f-4fc1-9fc2-760a4db6a08a" (UID: "1acc8610-820f-4fc1-9fc2-760a4db6a08a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:23.906986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.906868 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9895b9-a592-4acf-a80a-3ef016179bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9895b9-a592-4acf-a80a-3ef016179bc2-istiod-ca-cert\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907044 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-kube-api-access-hf9ck\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-kube-api-access-hf9ck\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907132 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-envoy\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907189 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-socket\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907252 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-data\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907284 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-podinfo\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907330 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-token\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907443 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-certs\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.907641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.907506 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-credential-socket\") pod \"5b9895b9-a592-4acf-a80a-3ef016179bc2\" (UID: \"5b9895b9-a592-4acf-a80a-3ef016179bc2\") " Oct 31 03:56:23.908071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.908050 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.908179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.908075 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.908179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.908091 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.908179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.908105 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1acc8610-820f-4fc1-9fc2-760a4db6a08a-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.908179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.908120 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7n5s\" (UniqueName: \"kubernetes.io/projected/1acc8610-820f-4fc1-9fc2-760a4db6a08a-kube-api-access-g7n5s\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.909534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.909502 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-socket" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.909892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.909871 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9895b9-a592-4acf-a80a-3ef016179bc2-istiod-ca-cert" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:23.910753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.910722 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-credential-socket" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.910929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.910910 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-data" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.911487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.911455 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-certs" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.912933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.912884 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-kube-api-access-hf9ck" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "kube-api-access-hf9ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.914737 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.914711 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-token" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.916481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.916450 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-envoy" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.918069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:23.918042 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-podinfo" pod "5b9895b9-a592-4acf-a80a-3ef016179bc2" (UID: "5b9895b9-a592-4acf-a80a-3ef016179bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:24.009776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009742 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.009776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009773 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.010037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009788 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.010037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009797 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9895b9-a592-4acf-a80a-3ef016179bc2-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.010037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009808 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9ck\" (UniqueName: \"kubernetes.io/projected/5b9895b9-a592-4acf-a80a-3ef016179bc2-kube-api-access-hf9ck\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.010037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009818 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.010037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009826 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.010037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009834 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.010037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.009842 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9895b9-a592-4acf-a80a-3ef016179bc2-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.609251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.607053 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" event={"ID":"1acc8610-820f-4fc1-9fc2-760a4db6a08a","Type":"ContainerDied","Data":"709a73ce10e78000bfa6f1f26cc6c973d0aa142a2243d1a1a3968b4da9134f8e"} Oct 31 03:56:24.609251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.607150 5520 scope.go:122] "RemoveContainer" containerID="07ae77079bb80b262c7d4f201d0c4149124c933ce576629c0e5fab757c40174a" Oct 31 03:56:24.609251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.607364 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts" Oct 31 03:56:24.612327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.612287 5520 generic.go:356] "Generic (PLEG): container finished" podID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerID="0c893f22fa4710f8b4c26fbf783b6c2a992702086104c30e2e30fdd64d28a0f0" exitCode=0 Oct 31 03:56:24.612453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.612439 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" event={"ID":"33e15ae6-cdb8-46aa-b7b8-983b7c19920c","Type":"ContainerDied","Data":"0c893f22fa4710f8b4c26fbf783b6c2a992702086104c30e2e30fdd64d28a0f0"} Oct 31 03:56:24.615026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.614956 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" event={"ID":"5b9895b9-a592-4acf-a80a-3ef016179bc2","Type":"ContainerDied","Data":"097653e7b5cd869bae4223b8ad9955d5ea7fe88963244c028b08ff537d892ff2"} Oct 31 03:56:24.615026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.615000 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj" Oct 31 03:56:24.627989 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.627968 5520 scope.go:122] "RemoveContainer" containerID="03e588934e172c2c07286643fbcf4ea37bc5984af5978292ad545b1397442455" Oct 31 03:56:24.642441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.642404 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts"] Oct 31 03:56:24.646675 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.645788 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-blmts"] Oct 31 03:56:24.657676 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.657640 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj"] Oct 31 03:56:24.663172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.663138 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-zlwlj"] Oct 31 03:56:24.665399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.665368 5520 scope.go:122] "RemoveContainer" containerID="fd5512af9ddd7d0e190a73e8d52c1ea3876ada5d8e1aed7a6e72c3e76cc69f15" Oct 31 03:56:24.880027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.879858 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:56:24.925652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.925613 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j"] Oct 31 03:56:24.932049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932015 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-certs\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932099 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-podinfo\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932170 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-envoy\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932327 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-kube-api-access-m58l7\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-kube-api-access-m58l7\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932381 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-data\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932409 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istiod-ca-cert\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932456 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-credential-socket\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932528 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-token\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-token\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.932609 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.932559 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-socket\") pod \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\" (UID: \"33e15ae6-cdb8-46aa-b7b8-983b7c19920c\") " Oct 31 03:56:24.933541 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.933513 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-socket" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.933745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.933726 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-certs" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.934273 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.934250 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-data" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.937354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.937054 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-podinfo" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:24.938154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.938128 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istiod-ca-cert" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:24.938287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.938214 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-credential-socket" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.938287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.938278 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-envoy" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.946741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.946195 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52"] Oct 31 03:56:24.953881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.952553 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-kube-api-access-m58l7" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "kube-api-access-m58l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:24.955093 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.955008 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-token" pod "33e15ae6-cdb8-46aa-b7b8-983b7c19920c" (UID: "33e15ae6-cdb8-46aa-b7b8-983b7c19920c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:24.958229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:24.957104 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k"] Oct 31 03:56:25.034441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034403 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034433 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034441 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034444 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034453 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034462 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034471 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034479 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034489 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.034760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.034498 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m58l7\" (UniqueName: \"kubernetes.io/projected/33e15ae6-cdb8-46aa-b7b8-983b7c19920c-kube-api-access-m58l7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.623158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.623063 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" Oct 31 03:56:25.623158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.623083 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c" event={"ID":"33e15ae6-cdb8-46aa-b7b8-983b7c19920c","Type":"ContainerDied","Data":"1ece5f157c891ddfbf116fdc42ffed34799cdb3d48931cc2c0312882f34a1656"} Oct 31 03:56:25.623158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.623138 5520 scope.go:122] "RemoveContainer" containerID="0c893f22fa4710f8b4c26fbf783b6c2a992702086104c30e2e30fdd64d28a0f0" Oct 31 03:56:25.648556 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.648526 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c"] Oct 31 03:56:25.667886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.667851 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-5h24c"] Oct 31 03:56:25.919664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.919590 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" path="/var/lib/kubelet/pods/1acc8610-820f-4fc1-9fc2-760a4db6a08a/volumes" Oct 31 03:56:25.920349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.920331 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" path="/var/lib/kubelet/pods/33e15ae6-cdb8-46aa-b7b8-983b7c19920c/volumes" Oct 31 03:56:25.920974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:25.920958 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" path="/var/lib/kubelet/pods/5b9895b9-a592-4acf-a80a-3ef016179bc2/volumes" Oct 31 03:56:27.188845 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.188809 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:56:27.271658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.271578 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-kube-api-access-hxv2d\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-kube-api-access-hxv2d\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.271658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.271635 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istiod-ca-cert\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.271913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.271738 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-token\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-token\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.271913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.271817 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-podinfo\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.271913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.271849 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-data\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.271913 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.271900 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-envoy\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.272137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272065 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-socket\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.272137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272089 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istiod-ca-cert" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:27.272137 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272127 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-certs\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.272342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272161 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-credential-socket\") pod \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\" (UID: \"f2aec880-cd9a-46fb-99a6-4a51e8921c94\") " Oct 31 03:56:27.272491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272391 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-socket" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.272491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272427 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-data" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.272700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272611 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-certs" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.272700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272669 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-credential-socket" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.272998 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.272977 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.273110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.273002 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.273110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.273018 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.273110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.273031 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.273110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.273044 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.275294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.275262 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-envoy" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.275459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.275383 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-podinfo" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:27.275459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.275395 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-kube-api-access-hxv2d" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "kube-api-access-hxv2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.276685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.276663 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-token" pod "f2aec880-cd9a-46fb-99a6-4a51e8921c94" (UID: "f2aec880-cd9a-46fb-99a6-4a51e8921c94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.374932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.374896 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.374932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.374929 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.375165 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.374943 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxv2d\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-kube-api-access-hxv2d\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.375165 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.374954 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2aec880-cd9a-46fb-99a6-4a51e8921c94-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.643756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.643722 5520 generic.go:356] "Generic (PLEG): container finished" podID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerID="cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821" exitCode=0 Oct 31 03:56:27.643877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.643850 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" Oct 31 03:56:27.643994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.643851 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" event={"ID":"f2aec880-cd9a-46fb-99a6-4a51e8921c94","Type":"ContainerDied","Data":"cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821"} Oct 31 03:56:27.644123 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.644096 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql" event={"ID":"f2aec880-cd9a-46fb-99a6-4a51e8921c94","Type":"ContainerDied","Data":"3447455847a126c026a523c582344ab174c6d95caa00837e01dd955574d3ec47"} Oct 31 03:56:27.644185 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.644138 5520 scope.go:122] "RemoveContainer" containerID="cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821" Oct 31 03:56:27.646936 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.646914 5520 generic.go:356] "Generic (PLEG): container finished" podID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerID="e501340198ca63d97c2d278945a48721672830474f8b248e63919c70fcfc89dc" exitCode=0 Oct 31 03:56:27.647181 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.646991 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" event={"ID":"8c92f1df-97ea-4f0f-8b10-3c04f109f33c","Type":"ContainerDied","Data":"e501340198ca63d97c2d278945a48721672830474f8b248e63919c70fcfc89dc"} Oct 31 03:56:27.691685 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.690752 5520 scope.go:122] "RemoveContainer" containerID="6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7" Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.711496 5520 scope.go:122] "RemoveContainer" containerID="cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821" Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:27.712054 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821\": container with ID starting with cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821 not found: ID does not exist" containerID="cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821" Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.712093 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821"} err="failed to get container status \"cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821\": rpc error: code = NotFound desc = could not find container \"cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821\": container with ID starting with cb7c404124ec72133bed545c54a952e0c11f1611b6e5c47faa94163200043821 not found: ID does not exist" Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.712115 5520 scope.go:122] "RemoveContainer" containerID="6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7" Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.712507 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql"] Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:27.713274 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7\": container with ID starting with 6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7 not found: ID does not exist" containerID="6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7" Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.713306 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7"} err="failed to get container status \"6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7\": rpc error: code = NotFound desc = could not find container \"6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7\": container with ID starting with 6f040dc4c9d5d5d9dd256c38b23898a7ff7af6b366c909e4e6d172b6713017d7 not found: ID does not exist" Oct 31 03:56:27.714252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.713329 5520 scope.go:122] "RemoveContainer" containerID="1ff892b288093ad0817ce15551c3e5de79c51d8380a54883765a4d85252e3c53" Oct 31 03:56:27.719446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.719420 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-fb8ql"] Oct 31 03:56:27.898735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.898693 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl"] Oct 31 03:56:27.936939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.936903 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" path="/var/lib/kubelet/pods/f2aec880-cd9a-46fb-99a6-4a51e8921c94/volumes" Oct 31 03:56:27.940030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.940000 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:56:27.979192 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:27.976590 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:56:28.124282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.124241 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-data\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.124489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.124325 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-socket\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.124489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.124370 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-certs\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.124489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.124406 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-kube-api-access-dtljg\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-kube-api-access-dtljg\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.124489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.124466 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-credential-socket\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.126240 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126174 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5883cc3-a2f9-4082-bd89-f0940ae80531-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5883cc3-a2f9-4082-bd89-f0940ae80531-istiod-ca-cert\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.126384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126263 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-socket\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.126384 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126320 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-istio-token\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-istio-token\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.126500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126388 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be417875-9d21-454c-908f-d5b154479635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be417875-9d21-454c-908f-d5b154479635-istio-podinfo\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.126500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126429 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-socket" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.126500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126457 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-token\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-token\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.126645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126538 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be417875-9d21-454c-908f-d5b154479635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be417875-9d21-454c-908f-d5b154479635-istiod-ca-cert\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.126645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126596 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-podinfo\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.126749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126691 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-data\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.126819 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126799 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-credential-socket\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.126878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126857 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-envoy\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.126928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126906 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-kube-api-access-rqwm8\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-kube-api-access-rqwm8\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.126988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126943 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-certs\") pod \"be417875-9d21-454c-908f-d5b154479635\" (UID: \"be417875-9d21-454c-908f-d5b154479635\") " Oct 31 03:56:28.127037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.126991 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-envoy\") pod \"e5883cc3-a2f9-4082-bd89-f0940ae80531\" (UID: \"e5883cc3-a2f9-4082-bd89-f0940ae80531\") " Oct 31 03:56:28.128267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.128238 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.128520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.128494 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-data" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.128933 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.128915 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-data" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.129050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.129021 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5883cc3-a2f9-4082-bd89-f0940ae80531-istiod-ca-cert" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:28.129389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.129362 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-certs" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.133219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.132394 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-token" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:28.133219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.132592 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-socket" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.133219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.132676 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be417875-9d21-454c-908f-d5b154479635-istiod-ca-cert" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:28.133856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.133650 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-kube-api-access-rqwm8" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "kube-api-access-rqwm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:28.154222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.133977 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-certs" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.154222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.134428 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-envoy" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.154222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.134535 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-kube-api-access-dtljg" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "kube-api-access-dtljg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:28.154222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.134653 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-credential-socket" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.154222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.134758 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-credential-socket" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.154222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.151860 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-istio-token" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:28.155785 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.155724 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be417875-9d21-454c-908f-d5b154479635-istio-podinfo" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:28.164403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.164350 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-envoy" pod "be417875-9d21-454c-908f-d5b154479635" (UID: "be417875-9d21-454c-908f-d5b154479635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.196701 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.187149 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-podinfo" pod "e5883cc3-a2f9-4082-bd89-f0940ae80531" (UID: "e5883cc3-a2f9-4082-bd89-f0940ae80531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243172 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5883cc3-a2f9-4082-bd89-f0940ae80531-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243228 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243246 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243259 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be417875-9d21-454c-908f-d5b154479635-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243273 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243287 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be417875-9d21-454c-908f-d5b154479635-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243300 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243314 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243328 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243343 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243358 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqwm8\" (UniqueName: \"kubernetes.io/projected/be417875-9d21-454c-908f-d5b154479635-kube-api-access-rqwm8\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243374 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243387 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243400 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be417875-9d21-454c-908f-d5b154479635-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243413 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243426 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtljg\" (UniqueName: \"kubernetes.io/projected/e5883cc3-a2f9-4082-bd89-f0940ae80531-kube-api-access-dtljg\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.243469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.243439 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5883cc3-a2f9-4082-bd89-f0940ae80531-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.355433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.354230 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:56:28.449051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449016 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-socket\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449113 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-data\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449149 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-envoy\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449273 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-token\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-token\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449345 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-kube-api-access-8rzqs\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-kube-api-access-8rzqs\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449355 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-socket" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.449437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449428 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-podinfo\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449437 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449424 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-data" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.449731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449530 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-credential-socket\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449600 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-certs\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449731 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449628 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istiod-ca-cert\") pod \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\" (UID: \"8c92f1df-97ea-4f0f-8b10-3c04f109f33c\") " Oct 31 03:56:28.449910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449799 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-credential-socket" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.449972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.449926 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-certs" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.450193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.450147 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istiod-ca-cert" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:28.450354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.450334 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.450354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.450352 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.450496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.450363 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.450496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.450372 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.450496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.450380 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.451567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.451549 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-token" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:28.451663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.451623 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-podinfo" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:28.452083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.452065 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-envoy" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:28.452163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.452096 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-kube-api-access-8rzqs" pod "8c92f1df-97ea-4f0f-8b10-3c04f109f33c" (UID: "8c92f1df-97ea-4f0f-8b10-3c04f109f33c"). InnerVolumeSpecName "kube-api-access-8rzqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:28.551363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.551328 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rzqs\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-kube-api-access-8rzqs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.551363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.551361 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.551363 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.551371 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.551626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.551380 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c92f1df-97ea-4f0f-8b10-3c04f109f33c-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.655946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.655867 5520 generic.go:356] "Generic (PLEG): container finished" podID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerID="6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a" exitCode=0 Oct 31 03:56:28.655946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.655929 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" Oct 31 03:56:28.656159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.655951 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" event={"ID":"e5883cc3-a2f9-4082-bd89-f0940ae80531","Type":"ContainerDied","Data":"6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a"} Oct 31 03:56:28.656159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.656037 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr" event={"ID":"e5883cc3-a2f9-4082-bd89-f0940ae80531","Type":"ContainerDied","Data":"0ba13b4045b2d3fee186d1b3f4018f29a006aeddb3b939ea45f3398d75920f7e"} Oct 31 03:56:28.656159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.656067 5520 scope.go:122] "RemoveContainer" containerID="6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a" Oct 31 03:56:28.657469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.657446 5520 generic.go:356] "Generic (PLEG): container finished" podID="be417875-9d21-454c-908f-d5b154479635" containerID="e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c" exitCode=0 Oct 31 03:56:28.657569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.657514 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" Oct 31 03:56:28.657637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.657577 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" event={"ID":"be417875-9d21-454c-908f-d5b154479635","Type":"ContainerDied","Data":"e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c"} Oct 31 03:56:28.657637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.657618 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs" event={"ID":"be417875-9d21-454c-908f-d5b154479635","Type":"ContainerDied","Data":"0269fb5874195eb75d93bec50dc3f1521547b278a1daa249462d8e2381f5ee02"} Oct 31 03:56:28.660245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.660222 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" event={"ID":"8c92f1df-97ea-4f0f-8b10-3c04f109f33c","Type":"ContainerDied","Data":"5e4ca64fd47881d3bac71900dbfac6b4a3b42c13643a3b015950f0baae38df89"} Oct 31 03:56:28.660385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.660230 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2" Oct 31 03:56:28.681042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.679278 5520 scope.go:122] "RemoveContainer" containerID="6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a" Oct 31 03:56:28.681360 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:28.681304 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a\": container with ID starting with 6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a not found: ID does not exist" containerID="6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a" Oct 31 03:56:28.681470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.681374 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a"} err="failed to get container status \"6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a\": rpc error: code = NotFound desc = could not find container \"6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a\": container with ID starting with 6ff240ac2026a421fd3c52adff944ab441b570d58c31b6b3c4560e2ab1c3488a not found: ID does not exist" Oct 31 03:56:28.681470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.681420 5520 scope.go:122] "RemoveContainer" containerID="e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c" Oct 31 03:56:28.702396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.702366 5520 scope.go:122] "RemoveContainer" containerID="e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c" Oct 31 03:56:28.702855 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:28.702828 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c\": container with ID starting with e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c not found: ID does not exist" containerID="e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c" Oct 31 03:56:28.702978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.702866 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c"} err="failed to get container status \"e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c\": rpc error: code = NotFound desc = could not find container \"e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c\": container with ID starting with e55598542ed077422088cf74a5b356de25b64d6c455e5ecc7134a2c77164090c not found: ID does not exist" Oct 31 03:56:28.702978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.702890 5520 scope.go:122] "RemoveContainer" containerID="e501340198ca63d97c2d278945a48721672830474f8b248e63919c70fcfc89dc" Oct 31 03:56:28.716857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.716814 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs"] Oct 31 03:56:28.719878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.719850 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-ld7rs"] Oct 31 03:56:28.729578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.729554 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2"] Oct 31 03:56:28.732013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.731994 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-fmnb2"] Oct 31 03:56:28.747732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.747682 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr"] Oct 31 03:56:28.755830 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.755807 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-hkpxr"] Oct 31 03:56:28.974760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:28.974728 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt"] Oct 31 03:56:29.920834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:29.920799 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" path="/var/lib/kubelet/pods/8c92f1df-97ea-4f0f-8b10-3c04f109f33c/volumes" Oct 31 03:56:29.921592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:29.921570 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be417875-9d21-454c-908f-d5b154479635" path="/var/lib/kubelet/pods/be417875-9d21-454c-908f-d5b154479635/volumes" Oct 31 03:56:29.922231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:29.922189 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" path="/var/lib/kubelet/pods/e5883cc3-a2f9-4082-bd89-f0940ae80531/volumes" Oct 31 03:56:30.414934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:30.414895 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf"] Oct 31 03:56:32.442807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:32.442775 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z"] Oct 31 03:56:32.480875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:32.480846 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv"] Oct 31 03:56:32.899494 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:32.899378 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" podUID="0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" containerName="payload-pre-processing" containerID="cri-o://b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b" gracePeriod=25 Oct 31 03:56:33.248482 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.248456 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:56:33.338631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.338603 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-kube-api-access-tv9ft\" (UniqueName: \"kubernetes.io/projected/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-kube-api-access-tv9ft\") pod \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " Oct 31 03:56:33.338812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.338738 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-plugins-config-volume\") pod \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\" (UID: \"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d\") " Oct 31 03:56:33.339705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.339678 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-plugins-config-volume" pod "0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" (UID: "0cc1a4d2-a402-470e-9dd2-31a9fb33be0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:33.342042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.342019 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-kube-api-access-tv9ft" pod "0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" (UID: "0cc1a4d2-a402-470e-9dd2-31a9fb33be0d"). InnerVolumeSpecName "kube-api-access-tv9ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:33.350593 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.350567 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh"] Oct 31 03:56:33.382606 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.382575 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk"] Oct 31 03:56:33.441339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.441305 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:33.441339 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.441335 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv9ft\" (UniqueName: \"kubernetes.io/projected/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d-kube-api-access-tv9ft\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:33.779229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.778820 5520 generic.go:356] "Generic (PLEG): container finished" podID="0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" containerID="b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b" exitCode=0 Oct 31 03:56:33.779229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.778962 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" event={"ID":"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d","Type":"ContainerDied","Data":"b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b"} Oct 31 03:56:33.779229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.779012 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" event={"ID":"0cc1a4d2-a402-470e-9dd2-31a9fb33be0d","Type":"ContainerDied","Data":"8266e503eace3aa7fa608596708e952c4baf44acec6cd6abc3ab1b52621cd4e5"} Oct 31 03:56:33.779229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.779041 5520 scope.go:122] "RemoveContainer" containerID="b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b" Oct 31 03:56:33.779229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.779181 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl" Oct 31 03:56:33.835022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.834723 5520 scope.go:122] "RemoveContainer" containerID="b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b" Oct 31 03:56:33.835656 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:33.835569 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b\": container with ID starting with b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b not found: ID does not exist" containerID="b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b" Oct 31 03:56:33.835656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.835606 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b"} err="failed to get container status \"b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b\": rpc error: code = NotFound desc = could not find container \"b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b\": container with ID starting with b0c290d2caa66ad214ff6bb862754293b8aaaa1781a047224c57f9742e05ae2b not found: ID does not exist" Oct 31 03:56:33.880242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.877953 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl"] Oct 31 03:56:33.882056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.881681 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-dda5cb-9cf74ccd7-89jgl"] Oct 31 03:56:33.952589 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.952552 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" path="/var/lib/kubelet/pods/0cc1a4d2-a402-470e-9dd2-31a9fb33be0d/volumes" Oct 31 03:56:33.977725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:33.977677 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" podUID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" containerName="payload-processing" containerID="cri-o://ff5f99dd04f716ea631432f027e0f47347cbe1cd1d6e0cae09ce79622afa0c8a" gracePeriod=25 Oct 31 03:56:34.041736 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.041644 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" containerID="cri-o://f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9" gracePeriod=45 Oct 31 03:56:34.330611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.330585 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:56:34.387827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.386512 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-dshm\") pod \"78562ae3-0ada-42c7-917f-52c0152b890a\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " Oct 31 03:56:34.387827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.386580 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-home\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-home\") pod \"78562ae3-0ada-42c7-917f-52c0152b890a\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " Oct 31 03:56:34.387827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.386664 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-tmp-dir\") pod \"78562ae3-0ada-42c7-917f-52c0152b890a\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " Oct 31 03:56:34.387827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.386732 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78562ae3-0ada-42c7-917f-52c0152b890a-tls-certs\" (UniqueName: \"kubernetes.io/secret/78562ae3-0ada-42c7-917f-52c0152b890a-tls-certs\") pod \"78562ae3-0ada-42c7-917f-52c0152b890a\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " Oct 31 03:56:34.387827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.386795 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-model-cache\") pod \"78562ae3-0ada-42c7-917f-52c0152b890a\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " Oct 31 03:56:34.387827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.386826 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78562ae3-0ada-42c7-917f-52c0152b890a-kube-api-access-25mhc\" (UniqueName: \"kubernetes.io/projected/78562ae3-0ada-42c7-917f-52c0152b890a-kube-api-access-25mhc\") pod \"78562ae3-0ada-42c7-917f-52c0152b890a\" (UID: \"78562ae3-0ada-42c7-917f-52c0152b890a\") " Oct 31 03:56:34.389986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.389919 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-tmp-dir" pod "78562ae3-0ada-42c7-917f-52c0152b890a" (UID: "78562ae3-0ada-42c7-917f-52c0152b890a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.391144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.390188 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-home" pod "78562ae3-0ada-42c7-917f-52c0152b890a" (UID: "78562ae3-0ada-42c7-917f-52c0152b890a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.391144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.390440 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-model-cache" pod "78562ae3-0ada-42c7-917f-52c0152b890a" (UID: "78562ae3-0ada-42c7-917f-52c0152b890a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.398275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.393508 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-dshm" pod "78562ae3-0ada-42c7-917f-52c0152b890a" (UID: "78562ae3-0ada-42c7-917f-52c0152b890a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.407127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.407096 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78562ae3-0ada-42c7-917f-52c0152b890a-tls-certs" pod "78562ae3-0ada-42c7-917f-52c0152b890a" (UID: "78562ae3-0ada-42c7-917f-52c0152b890a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:34.413894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.412361 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78562ae3-0ada-42c7-917f-52c0152b890a-kube-api-access-25mhc" pod "78562ae3-0ada-42c7-917f-52c0152b890a" (UID: "78562ae3-0ada-42c7-917f-52c0152b890a"). InnerVolumeSpecName "kube-api-access-25mhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:34.489294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.489257 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.489294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.489292 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.489531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.489306 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.489531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.489321 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78562ae3-0ada-42c7-917f-52c0152b890a-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.489531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.489334 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78562ae3-0ada-42c7-917f-52c0152b890a-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.489531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.489350 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25mhc\" (UniqueName: \"kubernetes.io/projected/78562ae3-0ada-42c7-917f-52c0152b890a-kube-api-access-25mhc\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.787044 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.786999 5520 generic.go:356] "Generic (PLEG): container finished" podID="78562ae3-0ada-42c7-917f-52c0152b890a" containerID="f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9" exitCode=0 Oct 31 03:56:34.787590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.787058 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" Oct 31 03:56:34.787590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.787096 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" event={"ID":"78562ae3-0ada-42c7-917f-52c0152b890a","Type":"ContainerDied","Data":"f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9"} Oct 31 03:56:34.787590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.787142 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98" event={"ID":"78562ae3-0ada-42c7-917f-52c0152b890a","Type":"ContainerDied","Data":"ce85d064437387d8925ce9f914b939752f86ad3db7239e0278f54f032e3c94cb"} Oct 31 03:56:34.787590 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.787165 5520 scope.go:122] "RemoveContainer" containerID="f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9" Oct 31 03:56:34.798095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.798071 5520 scope.go:122] "RemoveContainer" containerID="f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9" Oct 31 03:56:34.798448 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:34.798420 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9\": container with ID starting with f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9 not found: ID does not exist" containerID="f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9" Oct 31 03:56:34.798568 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.798455 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9"} err="failed to get container status \"f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9\": rpc error: code = NotFound desc = could not find container \"f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9\": container with ID starting with f59570ba334bdb2a2321643bdb76f84599e550500941d0a737fee2b9d22458f9 not found: ID does not exist" Oct 31 03:56:34.817729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.817687 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98"] Oct 31 03:56:34.820289 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:34.820268 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/premium-w0-d86a77-kserve-5496ccf66c-r7m98"] Oct 31 03:56:35.254327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.254288 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc"] Oct 31 03:56:35.262687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.260291 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" podUID="82b5e11c-6318-46e4-aa59-fb0d7b972691" containerName="maas-api" containerID="cri-o://c687756912fdbb040f60813c0c15fe85b64745df2ae2e8ac5b09d65a78dbffa0" gracePeriod=30 Oct 31 03:56:35.415435 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.415365 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" podUID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" containerName="payload-processing" containerID="cri-o://578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12" gracePeriod=25 Oct 31 03:56:35.801897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.801864 5520 generic.go:356] "Generic (PLEG): container finished" podID="82b5e11c-6318-46e4-aa59-fb0d7b972691" containerID="c687756912fdbb040f60813c0c15fe85b64745df2ae2e8ac5b09d65a78dbffa0" exitCode=0 Oct 31 03:56:35.802419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.801939 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" event={"ID":"82b5e11c-6318-46e4-aa59-fb0d7b972691","Type":"ContainerDied","Data":"c687756912fdbb040f60813c0c15fe85b64745df2ae2e8ac5b09d65a78dbffa0"} Oct 31 03:56:35.828275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.828245 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:56:35.919161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.919127 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b5e11c-6318-46e4-aa59-fb0d7b972691-kube-api-access-qklcv\" (UniqueName: \"kubernetes.io/projected/82b5e11c-6318-46e4-aa59-fb0d7b972691-kube-api-access-qklcv\") pod \"82b5e11c-6318-46e4-aa59-fb0d7b972691\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " Oct 31 03:56:35.919434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.919272 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-maas-api-tls\") pod \"82b5e11c-6318-46e4-aa59-fb0d7b972691\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " Oct 31 03:56:35.919434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.919341 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-metrics-certs\") pod \"82b5e11c-6318-46e4-aa59-fb0d7b972691\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " Oct 31 03:56:35.919567 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.919447 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b5e11c-6318-46e4-aa59-fb0d7b972691-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b5e11c-6318-46e4-aa59-fb0d7b972691-openshift-service-ca\") pod \"82b5e11c-6318-46e4-aa59-fb0d7b972691\" (UID: \"82b5e11c-6318-46e4-aa59-fb0d7b972691\") " Oct 31 03:56:35.921242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.920596 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b5e11c-6318-46e4-aa59-fb0d7b972691-openshift-service-ca" pod "82b5e11c-6318-46e4-aa59-fb0d7b972691" (UID: "82b5e11c-6318-46e4-aa59-fb0d7b972691"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:35.921947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.921916 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" path="/var/lib/kubelet/pods/78562ae3-0ada-42c7-917f-52c0152b890a/volumes" Oct 31 03:56:35.923362 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.923316 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-maas-api-tls" pod "82b5e11c-6318-46e4-aa59-fb0d7b972691" (UID: "82b5e11c-6318-46e4-aa59-fb0d7b972691"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:35.924735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.924710 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-metrics-certs" pod "82b5e11c-6318-46e4-aa59-fb0d7b972691" (UID: "82b5e11c-6318-46e4-aa59-fb0d7b972691"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:35.924835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:35.924804 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b5e11c-6318-46e4-aa59-fb0d7b972691-kube-api-access-qklcv" pod "82b5e11c-6318-46e4-aa59-fb0d7b972691" (UID: "82b5e11c-6318-46e4-aa59-fb0d7b972691"). InnerVolumeSpecName "kube-api-access-qklcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:36.021127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.021036 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:36.021127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.021069 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82b5e11c-6318-46e4-aa59-fb0d7b972691-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:36.021127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.021084 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b5e11c-6318-46e4-aa59-fb0d7b972691-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:36.021127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.021097 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklcv\" (UniqueName: \"kubernetes.io/projected/82b5e11c-6318-46e4-aa59-fb0d7b972691-kube-api-access-qklcv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:36.295534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.295443 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" podUID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:56:36.812396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.812350 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" Oct 31 03:56:36.812396 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.812373 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc" event={"ID":"82b5e11c-6318-46e4-aa59-fb0d7b972691","Type":"ContainerDied","Data":"65b84e1b277943cc695a7ee25f01abd98e5f865a0cf8bb9af45813eb60233e97"} Oct 31 03:56:36.812932 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.812447 5520 scope.go:122] "RemoveContainer" containerID="c687756912fdbb040f60813c0c15fe85b64745df2ae2e8ac5b09d65a78dbffa0" Oct 31 03:56:36.835689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.835655 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc"] Oct 31 03:56:36.838443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:36.838404 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fc5ece-7485f74b7-rqhjc"] Oct 31 03:56:37.439874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.439840 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p"] Oct 31 03:56:37.440111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.440050 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" podUID="aa02636b-059a-434e-b5a0-5d93f8313dc3" containerName="maas-api" containerID="cri-o://8646ee0c6a2302dbb7a4df8d7a9ff6deb0e01ae0ed8bc63c899b63c54a96d119" gracePeriod=30 Oct 31 03:56:37.443232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.443172 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" podUID="f25e37b8-2e60-4c2a-bc75-5a9e133a8269" containerName="payload-processing" containerID="cri-o://158a89e5807deddf646398c335a860717564b472ace0481843e121f835586b9b" gracePeriod=25 Oct 31 03:56:37.482163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.482108 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" podUID="c1381bbc-d77c-42c2-8ab7-d536c4251d1c" containerName="payload-pre-processing" containerID="cri-o://e738fa93fa58df2ace99e80db6173b201b0d4f4a343fe3561f9b2fe2347292cd" gracePeriod=25 Oct 31 03:56:37.576078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.572250 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" containerID="cri-o://3251d6567de7a077d7fb08ec4bc0dc0f785a1b12fb8eec665233896ff068988b" gracePeriod=45 Oct 31 03:56:37.615144 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.614408 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" containerID="cri-o://060c9b0a7716add8af1731b281f69ced8edca1c6bad3946aeef16e4a1f1870b1" gracePeriod=45 Oct 31 03:56:37.708287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.698386 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 31 03:56:37.794789 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.789751 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" podUID="aa02636b-059a-434e-b5a0-5d93f8313dc3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.175:8443/health\": dial tcp 10.134.0.175:8443: connect: connection refused" Oct 31 03:56:37.845221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.840638 5520 generic.go:356] "Generic (PLEG): container finished" podID="c1381bbc-d77c-42c2-8ab7-d536c4251d1c" containerID="e738fa93fa58df2ace99e80db6173b201b0d4f4a343fe3561f9b2fe2347292cd" exitCode=0 Oct 31 03:56:37.845221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.840739 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" event={"ID":"c1381bbc-d77c-42c2-8ab7-d536c4251d1c","Type":"ContainerDied","Data":"e738fa93fa58df2ace99e80db6173b201b0d4f4a343fe3561f9b2fe2347292cd"} Oct 31 03:56:37.910488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.888350 5520 generic.go:356] "Generic (PLEG): container finished" podID="63146752-5307-488c-9217-8d41a8e28e29" containerID="3251d6567de7a077d7fb08ec4bc0dc0f785a1b12fb8eec665233896ff068988b" exitCode=0 Oct 31 03:56:37.910488 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:37.888574 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" event={"ID":"63146752-5307-488c-9217-8d41a8e28e29","Type":"ContainerDied","Data":"3251d6567de7a077d7fb08ec4bc0dc0f785a1b12fb8eec665233896ff068988b"} Oct 31 03:56:38.021120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.021038 5520 generic.go:356] "Generic (PLEG): container finished" podID="f25e37b8-2e60-4c2a-bc75-5a9e133a8269" containerID="158a89e5807deddf646398c335a860717564b472ace0481843e121f835586b9b" exitCode=0 Oct 31 03:56:38.038910 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.038535 5520 generic.go:356] "Generic (PLEG): container finished" podID="aa02636b-059a-434e-b5a0-5d93f8313dc3" containerID="8646ee0c6a2302dbb7a4df8d7a9ff6deb0e01ae0ed8bc63c899b63c54a96d119" exitCode=0 Oct 31 03:56:38.044238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.044014 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b5e11c-6318-46e4-aa59-fb0d7b972691" path="/var/lib/kubelet/pods/82b5e11c-6318-46e4-aa59-fb0d7b972691/volumes" Oct 31 03:56:38.044758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.044479 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" event={"ID":"f25e37b8-2e60-4c2a-bc75-5a9e133a8269","Type":"ContainerDied","Data":"158a89e5807deddf646398c335a860717564b472ace0481843e121f835586b9b"} Oct 31 03:56:38.044758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.044539 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" event={"ID":"aa02636b-059a-434e-b5a0-5d93f8313dc3","Type":"ContainerDied","Data":"8646ee0c6a2302dbb7a4df8d7a9ff6deb0e01ae0ed8bc63c899b63c54a96d119"} Oct 31 03:56:38.051311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.050043 5520 generic.go:356] "Generic (PLEG): container finished" podID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerID="060c9b0a7716add8af1731b281f69ced8edca1c6bad3946aeef16e4a1f1870b1" exitCode=0 Oct 31 03:56:38.051311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.050121 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" event={"ID":"b1ba2dce-da62-4cea-bfce-7c77a1c6121d","Type":"ContainerDied","Data":"060c9b0a7716add8af1731b281f69ced8edca1c6bad3946aeef16e4a1f1870b1"} Oct 31 03:56:38.203312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.203261 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:56:38.220787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.220703 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:56:38.251574 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.251544 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:56:38.269410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.268826 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:56:38.295420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.295141 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-maas-api-tls\") pod \"aa02636b-059a-434e-b5a0-5d93f8313dc3\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " Oct 31 03:56:38.296430 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.296391 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa02636b-059a-434e-b5a0-5d93f8313dc3-kube-api-access-ks7cd\" (UniqueName: \"kubernetes.io/projected/aa02636b-059a-434e-b5a0-5d93f8313dc3-kube-api-access-ks7cd\") pod \"aa02636b-059a-434e-b5a0-5d93f8313dc3\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " Oct 31 03:56:38.296611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.296587 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa02636b-059a-434e-b5a0-5d93f8313dc3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa02636b-059a-434e-b5a0-5d93f8313dc3-openshift-service-ca\") pod \"aa02636b-059a-434e-b5a0-5d93f8313dc3\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " Oct 31 03:56:38.296759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.296744 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-metrics-certs\") pod \"aa02636b-059a-434e-b5a0-5d93f8313dc3\" (UID: \"aa02636b-059a-434e-b5a0-5d93f8313dc3\") " Oct 31 03:56:38.296999 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.296977 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-plugins-config-volume\") pod \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " Oct 31 03:56:38.297334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.297316 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-kube-api-access-kbhxd\" (UniqueName: \"kubernetes.io/projected/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-kube-api-access-kbhxd\") pod \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\" (UID: \"f25e37b8-2e60-4c2a-bc75-5a9e133a8269\") " Oct 31 03:56:38.298900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.298727 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-plugins-config-volume" pod "f25e37b8-2e60-4c2a-bc75-5a9e133a8269" (UID: "f25e37b8-2e60-4c2a-bc75-5a9e133a8269"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:38.298900 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.298829 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa02636b-059a-434e-b5a0-5d93f8313dc3-openshift-service-ca" pod "aa02636b-059a-434e-b5a0-5d93f8313dc3" (UID: "aa02636b-059a-434e-b5a0-5d93f8313dc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:38.299421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.299373 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.299421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.299400 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa02636b-059a-434e-b5a0-5d93f8313dc3-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.306121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.305848 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa02636b-059a-434e-b5a0-5d93f8313dc3-kube-api-access-ks7cd" pod "aa02636b-059a-434e-b5a0-5d93f8313dc3" (UID: "aa02636b-059a-434e-b5a0-5d93f8313dc3"). InnerVolumeSpecName "kube-api-access-ks7cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.306922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.306880 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-maas-api-tls" pod "aa02636b-059a-434e-b5a0-5d93f8313dc3" (UID: "aa02636b-059a-434e-b5a0-5d93f8313dc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:38.308222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.308171 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-metrics-certs" pod "aa02636b-059a-434e-b5a0-5d93f8313dc3" (UID: "aa02636b-059a-434e-b5a0-5d93f8313dc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:38.310035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.310012 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-kube-api-access-kbhxd" pod "f25e37b8-2e60-4c2a-bc75-5a9e133a8269" (UID: "f25e37b8-2e60-4c2a-bc75-5a9e133a8269"). InnerVolumeSpecName "kube-api-access-kbhxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.351477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.351433 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" podUID="7e1bca48-0aeb-42b6-b774-240be3e897e9" containerName="payload-processing" containerID="cri-o://e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617" gracePeriod=25 Oct 31 03:56:38.383715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.383655 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" podUID="089d79dd-f7ba-478f-9bb8-701683e36f62" containerName="payload-pre-processing" containerID="cri-o://eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97" gracePeriod=25 Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401300 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-plugins-config-volume\") pod \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401405 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-tmp-dir\") pod \"63146752-5307-488c-9217-8d41a8e28e29\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401505 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63146752-5307-488c-9217-8d41a8e28e29-kube-api-access-d8dcw\" (UniqueName: \"kubernetes.io/projected/63146752-5307-488c-9217-8d41a8e28e29-kube-api-access-d8dcw\") pod \"63146752-5307-488c-9217-8d41a8e28e29\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401547 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63146752-5307-488c-9217-8d41a8e28e29-tls-certs\" (UniqueName: \"kubernetes.io/secret/63146752-5307-488c-9217-8d41a8e28e29-tls-certs\") pod \"63146752-5307-488c-9217-8d41a8e28e29\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401615 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-kube-api-access-vvdh9\" (UniqueName: \"kubernetes.io/projected/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-kube-api-access-vvdh9\") pod \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\" (UID: \"c1381bbc-d77c-42c2-8ab7-d536c4251d1c\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401730 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-model-cache\") pod \"63146752-5307-488c-9217-8d41a8e28e29\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401834 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-home\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-home\") pod \"63146752-5307-488c-9217-8d41a8e28e29\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.401858 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-dshm\") pod \"63146752-5307-488c-9217-8d41a8e28e29\" (UID: \"63146752-5307-488c-9217-8d41a8e28e29\") " Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.406098 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-home" pod "63146752-5307-488c-9217-8d41a8e28e29" (UID: "63146752-5307-488c-9217-8d41a8e28e29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.406164 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhxd\" (UniqueName: \"kubernetes.io/projected/f25e37b8-2e60-4c2a-bc75-5a9e133a8269-kube-api-access-kbhxd\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.406177 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-model-cache" pod "63146752-5307-488c-9217-8d41a8e28e29" (UID: "63146752-5307-488c-9217-8d41a8e28e29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.406188 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.406324 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks7cd\" (UniqueName: \"kubernetes.io/projected/aa02636b-059a-434e-b5a0-5d93f8313dc3-kube-api-access-ks7cd\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.406398 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa02636b-059a-434e-b5a0-5d93f8313dc3-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.407087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.406567 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-plugins-config-volume" pod "c1381bbc-d77c-42c2-8ab7-d536c4251d1c" (UID: "c1381bbc-d77c-42c2-8ab7-d536c4251d1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:38.408138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.408113 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-tmp-dir" pod "63146752-5307-488c-9217-8d41a8e28e29" (UID: "63146752-5307-488c-9217-8d41a8e28e29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.409003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.408975 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63146752-5307-488c-9217-8d41a8e28e29-kube-api-access-d8dcw" pod "63146752-5307-488c-9217-8d41a8e28e29" (UID: "63146752-5307-488c-9217-8d41a8e28e29"). InnerVolumeSpecName "kube-api-access-d8dcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.409108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.409022 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63146752-5307-488c-9217-8d41a8e28e29-tls-certs" pod "63146752-5307-488c-9217-8d41a8e28e29" (UID: "63146752-5307-488c-9217-8d41a8e28e29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:38.410058 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.410039 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-dshm" pod "63146752-5307-488c-9217-8d41a8e28e29" (UID: "63146752-5307-488c-9217-8d41a8e28e29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.410537 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.410522 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-kube-api-access-vvdh9" pod "c1381bbc-d77c-42c2-8ab7-d536c4251d1c" (UID: "c1381bbc-d77c-42c2-8ab7-d536c4251d1c"). InnerVolumeSpecName "kube-api-access-vvdh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.441015 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.440993 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:56:38.512345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.512185 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-kube-api-access-9fxl6\" (UniqueName: \"kubernetes.io/projected/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-kube-api-access-9fxl6\") pod \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " Oct 31 03:56:38.512345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.512298 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tls-certs\") pod \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " Oct 31 03:56:38.512562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.512345 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-home\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-home\") pod \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " Oct 31 03:56:38.512562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.512436 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-dshm\") pod \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " Oct 31 03:56:38.512562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.512484 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tmp-dir\") pod \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " Oct 31 03:56:38.512801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.512663 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-model-cache\") pod \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\" (UID: \"b1ba2dce-da62-4cea-bfce-7c77a1c6121d\") " Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513179 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513227 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513241 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513254 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513266 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63146752-5307-488c-9217-8d41a8e28e29-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513278 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dcw\" (UniqueName: \"kubernetes.io/projected/63146752-5307-488c-9217-8d41a8e28e29-kube-api-access-d8dcw\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513290 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63146752-5307-488c-9217-8d41a8e28e29-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513303 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvdh9\" (UniqueName: \"kubernetes.io/projected/c1381bbc-d77c-42c2-8ab7-d536c4251d1c-kube-api-access-vvdh9\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.513803 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-home" pod "b1ba2dce-da62-4cea-bfce-7c77a1c6121d" (UID: "b1ba2dce-da62-4cea-bfce-7c77a1c6121d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.514756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.514459 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-model-cache" pod "b1ba2dce-da62-4cea-bfce-7c77a1c6121d" (UID: "b1ba2dce-da62-4cea-bfce-7c77a1c6121d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.515894 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.515844 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tmp-dir" pod "b1ba2dce-da62-4cea-bfce-7c77a1c6121d" (UID: "b1ba2dce-da62-4cea-bfce-7c77a1c6121d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.521786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.521606 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-dshm" pod "b1ba2dce-da62-4cea-bfce-7c77a1c6121d" (UID: "b1ba2dce-da62-4cea-bfce-7c77a1c6121d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.522870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.522657 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tls-certs" pod "b1ba2dce-da62-4cea-bfce-7c77a1c6121d" (UID: "b1ba2dce-da62-4cea-bfce-7c77a1c6121d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:38.524596 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.524569 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-kube-api-access-9fxl6" pod "b1ba2dce-da62-4cea-bfce-7c77a1c6121d" (UID: "b1ba2dce-da62-4cea-bfce-7c77a1c6121d"). InnerVolumeSpecName "kube-api-access-9fxl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.614380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.614349 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.614518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.614384 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.614518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.614400 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.614518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.614414 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fxl6\" (UniqueName: \"kubernetes.io/projected/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-kube-api-access-9fxl6\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.614518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.614427 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.614518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.614441 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1ba2dce-da62-4cea-bfce-7c77a1c6121d-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.709706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.709538 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:56:38.732717 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.732691 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:56:38.829732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.829170 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089d79dd-f7ba-478f-9bb8-701683e36f62-kube-api-access-wlpz8\" (UniqueName: \"kubernetes.io/projected/089d79dd-f7ba-478f-9bb8-701683e36f62-kube-api-access-wlpz8\") pod \"089d79dd-f7ba-478f-9bb8-701683e36f62\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " Oct 31 03:56:38.829732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.829363 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089d79dd-f7ba-478f-9bb8-701683e36f62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d79dd-f7ba-478f-9bb8-701683e36f62-plugins-config-volume\") pod \"089d79dd-f7ba-478f-9bb8-701683e36f62\" (UID: \"089d79dd-f7ba-478f-9bb8-701683e36f62\") " Oct 31 03:56:38.829732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.829637 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e1bca48-0aeb-42b6-b774-240be3e897e9-kube-api-access-7mznw\" (UniqueName: \"kubernetes.io/projected/7e1bca48-0aeb-42b6-b774-240be3e897e9-kube-api-access-7mznw\") pod \"7e1bca48-0aeb-42b6-b774-240be3e897e9\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " Oct 31 03:56:38.829732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.829695 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e1bca48-0aeb-42b6-b774-240be3e897e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1bca48-0aeb-42b6-b774-240be3e897e9-plugins-config-volume\") pod \"7e1bca48-0aeb-42b6-b774-240be3e897e9\" (UID: \"7e1bca48-0aeb-42b6-b774-240be3e897e9\") " Oct 31 03:56:38.830615 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.830589 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089d79dd-f7ba-478f-9bb8-701683e36f62-plugins-config-volume" pod "089d79dd-f7ba-478f-9bb8-701683e36f62" (UID: "089d79dd-f7ba-478f-9bb8-701683e36f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:38.831421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.831384 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1bca48-0aeb-42b6-b774-240be3e897e9-plugins-config-volume" pod "7e1bca48-0aeb-42b6-b774-240be3e897e9" (UID: "7e1bca48-0aeb-42b6-b774-240be3e897e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:38.835877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.835840 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e1bca48-0aeb-42b6-b774-240be3e897e9-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.835993 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.835882 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d79dd-f7ba-478f-9bb8-701683e36f62-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.837187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.837138 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089d79dd-f7ba-478f-9bb8-701683e36f62-kube-api-access-wlpz8" pod "089d79dd-f7ba-478f-9bb8-701683e36f62" (UID: "089d79dd-f7ba-478f-9bb8-701683e36f62"). InnerVolumeSpecName "kube-api-access-wlpz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.845837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.845742 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1bca48-0aeb-42b6-b774-240be3e897e9-kube-api-access-7mznw" pod "7e1bca48-0aeb-42b6-b774-240be3e897e9" (UID: "7e1bca48-0aeb-42b6-b774-240be3e897e9"). InnerVolumeSpecName "kube-api-access-7mznw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.936982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.936942 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mznw\" (UniqueName: \"kubernetes.io/projected/7e1bca48-0aeb-42b6-b774-240be3e897e9-kube-api-access-7mznw\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.936982 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:38.936975 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlpz8\" (UniqueName: \"kubernetes.io/projected/089d79dd-f7ba-478f-9bb8-701683e36f62-kube-api-access-wlpz8\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:39.057778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.057748 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" Oct 31 03:56:39.057995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.057747 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv" event={"ID":"c1381bbc-d77c-42c2-8ab7-d536c4251d1c","Type":"ContainerDied","Data":"a3a6a318227d78d81465fff65d3a2f41959ad0aa4455c44d995a92415133d80e"} Oct 31 03:56:39.057995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.057922 5520 scope.go:122] "RemoveContainer" containerID="e738fa93fa58df2ace99e80db6173b201b0d4f4a343fe3561f9b2fe2347292cd" Oct 31 03:56:39.059431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.059405 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" event={"ID":"63146752-5307-488c-9217-8d41a8e28e29","Type":"ContainerDied","Data":"b1c00053ce42a38baaedee2cc5e2ba2e510f7778fdeffaded848af08ceda24f2"} Oct 31 03:56:39.059512 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.059485 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx" Oct 31 03:56:39.062184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.062163 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" Oct 31 03:56:39.062316 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.062186 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z" event={"ID":"f25e37b8-2e60-4c2a-bc75-5a9e133a8269","Type":"ContainerDied","Data":"7cd29e320df8e29aa33f99b2ef3e7a48a6021a9aedbb0a36a1ef6a1b96ddfb8a"} Oct 31 03:56:39.063878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.063854 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" event={"ID":"aa02636b-059a-434e-b5a0-5d93f8313dc3","Type":"ContainerDied","Data":"0f52ec69a8b664a9ff8083df23996f23845f1cef6685898bf2fe373c2dbbee28"} Oct 31 03:56:39.063974 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.063936 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p" Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.066706 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" event={"ID":"b1ba2dce-da62-4cea-bfce-7c77a1c6121d","Type":"ContainerDied","Data":"363ef84774cf61cd206ae19ac402de9f27e1072074a82e3e58defbee53f66220"} Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.066731 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf" Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.068857 5520 generic.go:356] "Generic (PLEG): container finished" podID="7e1bca48-0aeb-42b6-b774-240be3e897e9" containerID="e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617" exitCode=0 Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.068957 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" event={"ID":"7e1bca48-0aeb-42b6-b774-240be3e897e9","Type":"ContainerDied","Data":"e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617"} Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.068991 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" event={"ID":"7e1bca48-0aeb-42b6-b774-240be3e897e9","Type":"ContainerDied","Data":"969235c7d05d7f8292045744f6edf433d6f66aca4affcc050b31362843b5da4b"} Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.068992 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh" Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.070393 5520 generic.go:356] "Generic (PLEG): container finished" podID="089d79dd-f7ba-478f-9bb8-701683e36f62" containerID="eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97" exitCode=0 Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.070424 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.070431 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" event={"ID":"089d79dd-f7ba-478f-9bb8-701683e36f62","Type":"ContainerDied","Data":"eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97"} Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.070464 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk" event={"ID":"089d79dd-f7ba-478f-9bb8-701683e36f62","Type":"ContainerDied","Data":"16ece96c6471fc0520bfae0c99b85bdfd994deccaf459a689f4bcea387c7a695"} Oct 31 03:56:39.085065 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.074320 5520 scope.go:122] "RemoveContainer" containerID="3251d6567de7a077d7fb08ec4bc0dc0f785a1b12fb8eec665233896ff068988b" Oct 31 03:56:39.087617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.087580 5520 scope.go:122] "RemoveContainer" containerID="158a89e5807deddf646398c335a860717564b472ace0481843e121f835586b9b" Oct 31 03:56:39.100929 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.100906 5520 scope.go:122] "RemoveContainer" containerID="8646ee0c6a2302dbb7a4df8d7a9ff6deb0e01ae0ed8bc63c899b63c54a96d119" Oct 31 03:56:39.107461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.107435 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv"] Oct 31 03:56:39.111653 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.111625 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fc5ece-64f6cc58d7-s2thv"] Oct 31 03:56:39.113108 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.113089 5520 scope.go:122] "RemoveContainer" containerID="060c9b0a7716add8af1731b281f69ced8edca1c6bad3946aeef16e4a1f1870b1" Oct 31 03:56:39.122965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.122948 5520 scope.go:122] "RemoveContainer" containerID="e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617" Oct 31 03:56:39.127849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.127829 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx"] Oct 31 03:56:39.132511 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.132489 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-w4-dda5cb-kserve-cdb6b8946-98jjx"] Oct 31 03:56:39.137949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.137391 5520 scope.go:122] "RemoveContainer" containerID="e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617" Oct 31 03:56:39.138637 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:39.138609 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617\": container with ID starting with e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617 not found: ID does not exist" containerID="e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617" Oct 31 03:56:39.138710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.138648 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617"} err="failed to get container status \"e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617\": rpc error: code = NotFound desc = could not find container \"e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617\": container with ID starting with e2f893fef523fc87aaac0613c84040d1f9422a84647a45bd6db874b1dd3e4617 not found: ID does not exist" Oct 31 03:56:39.138710 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.138674 5520 scope.go:122] "RemoveContainer" containerID="eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97" Oct 31 03:56:39.149049 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.149017 5520 scope.go:122] "RemoveContainer" containerID="eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97" Oct 31 03:56:39.149395 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:39.149360 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97\": container with ID starting with eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97 not found: ID does not exist" containerID="eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97" Oct 31 03:56:39.149489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.149404 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97"} err="failed to get container status \"eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97\": rpc error: code = NotFound desc = could not find container \"eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97\": container with ID starting with eac5f9fe3f1cb4365cdd4925bc12e7208d7dd5c9ebee23b67b6527e49f2a5c97 not found: ID does not exist" Oct 31 03:56:39.150988 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.150968 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p"] Oct 31 03:56:39.154432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.154412 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-86a40d-58b65f588d-bk87p"] Oct 31 03:56:39.167160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.167129 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z"] Oct 31 03:56:39.172524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.172497 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fc5ece-8b78b6b59-4646z"] Oct 31 03:56:39.181975 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.181944 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh"] Oct 31 03:56:39.184184 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.184165 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d04aad-c4fc7674f-wdfmh"] Oct 31 03:56:39.194465 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.194439 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf"] Oct 31 03:56:39.202083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.202055 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/simulator-w4-dda5cb-kserve-d7fcbfb4b-xmpmf"] Oct 31 03:56:39.212691 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.212598 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk"] Oct 31 03:56:39.216173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.216142 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d04aad-54fc84474c-nnhtk"] Oct 31 03:56:39.359893 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:39.359808 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:56:39.919161 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.919126 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089d79dd-f7ba-478f-9bb8-701683e36f62" path="/var/lib/kubelet/pods/089d79dd-f7ba-478f-9bb8-701683e36f62/volumes" Oct 31 03:56:39.919812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.919791 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63146752-5307-488c-9217-8d41a8e28e29" path="/var/lib/kubelet/pods/63146752-5307-488c-9217-8d41a8e28e29/volumes" Oct 31 03:56:39.920296 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.920276 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e1bca48-0aeb-42b6-b774-240be3e897e9" path="/var/lib/kubelet/pods/7e1bca48-0aeb-42b6-b774-240be3e897e9/volumes" Oct 31 03:56:39.921014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.920993 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa02636b-059a-434e-b5a0-5d93f8313dc3" path="/var/lib/kubelet/pods/aa02636b-059a-434e-b5a0-5d93f8313dc3/volumes" Oct 31 03:56:39.921469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.921451 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" path="/var/lib/kubelet/pods/b1ba2dce-da62-4cea-bfce-7c77a1c6121d/volumes" Oct 31 03:56:39.921878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.921860 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1381bbc-d77c-42c2-8ab7-d536c4251d1c" path="/var/lib/kubelet/pods/c1381bbc-d77c-42c2-8ab7-d536c4251d1c/volumes" Oct 31 03:56:39.922279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.922263 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25e37b8-2e60-4c2a-bc75-5a9e133a8269" path="/var/lib/kubelet/pods/f25e37b8-2e60-4c2a-bc75-5a9e133a8269/volumes" Oct 31 03:56:39.969964 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.969917 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" containerID="cri-o://945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3" gracePeriod=45 Oct 31 03:56:39.978350 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.978302 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" containerID="cri-o://cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4" gracePeriod=45 Oct 31 03:56:39.980562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:39.980526 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" containerID="cri-o://5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c" gracePeriod=45 Oct 31 03:56:40.126712 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.126668 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused" Oct 31 03:56:40.261356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.260670 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:56:40.305267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.305241 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:56:40.313357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.313327 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:56:40.358842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.358793 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b46c5ba-21f9-4d91-adbc-16e654451c06-kube-api-access-8nflt\" (UniqueName: \"kubernetes.io/projected/3b46c5ba-21f9-4d91-adbc-16e654451c06-kube-api-access-8nflt\") pod \"3b46c5ba-21f9-4d91-adbc-16e654451c06\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " Oct 31 03:56:40.359224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.359186 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-model-cache\") pod \"3b46c5ba-21f9-4d91-adbc-16e654451c06\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " Oct 31 03:56:40.359411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.359379 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-tmp-dir\") pod \"3b46c5ba-21f9-4d91-adbc-16e654451c06\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " Oct 31 03:56:40.359604 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.359585 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-home\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-home\") pod \"3b46c5ba-21f9-4d91-adbc-16e654451c06\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " Oct 31 03:56:40.359793 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.359777 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b46c5ba-21f9-4d91-adbc-16e654451c06-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b46c5ba-21f9-4d91-adbc-16e654451c06-tls-certs\") pod \"3b46c5ba-21f9-4d91-adbc-16e654451c06\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " Oct 31 03:56:40.359956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.359927 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-dshm\") pod \"3b46c5ba-21f9-4d91-adbc-16e654451c06\" (UID: \"3b46c5ba-21f9-4d91-adbc-16e654451c06\") " Oct 31 03:56:40.361424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.361398 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-tmp-dir" pod "3b46c5ba-21f9-4d91-adbc-16e654451c06" (UID: "3b46c5ba-21f9-4d91-adbc-16e654451c06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.361636 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.361613 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-model-cache" pod "3b46c5ba-21f9-4d91-adbc-16e654451c06" (UID: "3b46c5ba-21f9-4d91-adbc-16e654451c06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.361878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.361837 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-home" pod "3b46c5ba-21f9-4d91-adbc-16e654451c06" (UID: "3b46c5ba-21f9-4d91-adbc-16e654451c06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.363487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.363455 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-dshm" pod "3b46c5ba-21f9-4d91-adbc-16e654451c06" (UID: "3b46c5ba-21f9-4d91-adbc-16e654451c06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.363954 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.363927 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b46c5ba-21f9-4d91-adbc-16e654451c06-kube-api-access-8nflt" pod "3b46c5ba-21f9-4d91-adbc-16e654451c06" (UID: "3b46c5ba-21f9-4d91-adbc-16e654451c06"). InnerVolumeSpecName "kube-api-access-8nflt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:40.365079 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.365056 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b46c5ba-21f9-4d91-adbc-16e654451c06-tls-certs" pod "3b46c5ba-21f9-4d91-adbc-16e654451c06" (UID: "3b46c5ba-21f9-4d91-adbc-16e654451c06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:40.461795 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.461698 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f184c11a-b290-49a2-bab7-0e3bad0dbddd-kube-api-access-rwzb8\" (UniqueName: \"kubernetes.io/projected/f184c11a-b290-49a2-bab7-0e3bad0dbddd-kube-api-access-rwzb8\") pod \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " Oct 31 03:56:40.461962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.461829 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/195e4171-ca09-4c7f-afbe-08a2fa33a736-tls-certs\" (UniqueName: \"kubernetes.io/secret/195e4171-ca09-4c7f-afbe-08a2fa33a736-tls-certs\") pod \"195e4171-ca09-4c7f-afbe-08a2fa33a736\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " Oct 31 03:56:40.461962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.461877 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-dshm\") pod \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " Oct 31 03:56:40.461962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.461911 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-dshm\") pod \"195e4171-ca09-4c7f-afbe-08a2fa33a736\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " Oct 31 03:56:40.461962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.461947 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-tmp-dir\") pod \"195e4171-ca09-4c7f-afbe-08a2fa33a736\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " Oct 31 03:56:40.462190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.461976 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-home\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-home\") pod \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " Oct 31 03:56:40.462190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462050 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tls-certs\" (UniqueName: \"kubernetes.io/secret/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tls-certs\") pod \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " Oct 31 03:56:40.462190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462098 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tmp-dir\") pod \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " Oct 31 03:56:40.462190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462128 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-model-cache\") pod \"195e4171-ca09-4c7f-afbe-08a2fa33a736\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " Oct 31 03:56:40.462190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462178 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-model-cache\") pod \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\" (UID: \"f184c11a-b290-49a2-bab7-0e3bad0dbddd\") " Oct 31 03:56:40.462474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462311 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-home\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-home\") pod \"195e4171-ca09-4c7f-afbe-08a2fa33a736\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " Oct 31 03:56:40.462474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462345 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195e4171-ca09-4c7f-afbe-08a2fa33a736-kube-api-access-sg67q\" (UniqueName: \"kubernetes.io/projected/195e4171-ca09-4c7f-afbe-08a2fa33a736-kube-api-access-sg67q\") pod \"195e4171-ca09-4c7f-afbe-08a2fa33a736\" (UID: \"195e4171-ca09-4c7f-afbe-08a2fa33a736\") " Oct 31 03:56:40.462474 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462427 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tmp-dir" pod "f184c11a-b290-49a2-bab7-0e3bad0dbddd" (UID: "f184c11a-b290-49a2-bab7-0e3bad0dbddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.462640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462569 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-home" pod "f184c11a-b290-49a2-bab7-0e3bad0dbddd" (UID: "f184c11a-b290-49a2-bab7-0e3bad0dbddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.462766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462712 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-model-cache" pod "f184c11a-b290-49a2-bab7-0e3bad0dbddd" (UID: "f184c11a-b290-49a2-bab7-0e3bad0dbddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.462909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462883 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b46c5ba-21f9-4d91-adbc-16e654451c06-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462913 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462927 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462939 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462952 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462967 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nflt\" (UniqueName: \"kubernetes.io/projected/3b46c5ba-21f9-4d91-adbc-16e654451c06-kube-api-access-8nflt\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462982 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.462997 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.463011 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b46c5ba-21f9-4d91-adbc-16e654451c06-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.463445 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.463263 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-model-cache" pod "195e4171-ca09-4c7f-afbe-08a2fa33a736" (UID: "195e4171-ca09-4c7f-afbe-08a2fa33a736"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.463602 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.463582 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-home" pod "195e4171-ca09-4c7f-afbe-08a2fa33a736" (UID: "195e4171-ca09-4c7f-afbe-08a2fa33a736"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.464867 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.464839 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-tmp-dir" pod "195e4171-ca09-4c7f-afbe-08a2fa33a736" (UID: "195e4171-ca09-4c7f-afbe-08a2fa33a736"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.464968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.464879 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tls-certs" pod "f184c11a-b290-49a2-bab7-0e3bad0dbddd" (UID: "f184c11a-b290-49a2-bab7-0e3bad0dbddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:40.464968 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.464893 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195e4171-ca09-4c7f-afbe-08a2fa33a736-tls-certs" pod "195e4171-ca09-4c7f-afbe-08a2fa33a736" (UID: "195e4171-ca09-4c7f-afbe-08a2fa33a736"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:40.465103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.465084 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f184c11a-b290-49a2-bab7-0e3bad0dbddd-kube-api-access-rwzb8" pod "f184c11a-b290-49a2-bab7-0e3bad0dbddd" (UID: "f184c11a-b290-49a2-bab7-0e3bad0dbddd"). InnerVolumeSpecName "kube-api-access-rwzb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:40.465424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.465400 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-dshm" pod "f184c11a-b290-49a2-bab7-0e3bad0dbddd" (UID: "f184c11a-b290-49a2-bab7-0e3bad0dbddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.465702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.465681 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195e4171-ca09-4c7f-afbe-08a2fa33a736-kube-api-access-sg67q" pod "195e4171-ca09-4c7f-afbe-08a2fa33a736" (UID: "195e4171-ca09-4c7f-afbe-08a2fa33a736"). InnerVolumeSpecName "kube-api-access-sg67q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:40.466450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.466434 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-dshm" pod "195e4171-ca09-4c7f-afbe-08a2fa33a736" (UID: "195e4171-ca09-4c7f-afbe-08a2fa33a736"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:40.564271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564194 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564267 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg67q\" (UniqueName: \"kubernetes.io/projected/195e4171-ca09-4c7f-afbe-08a2fa33a736-kube-api-access-sg67q\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564277 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwzb8\" (UniqueName: \"kubernetes.io/projected/f184c11a-b290-49a2-bab7-0e3bad0dbddd-kube-api-access-rwzb8\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564287 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195e4171-ca09-4c7f-afbe-08a2fa33a736-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564296 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f184c11a-b290-49a2-bab7-0e3bad0dbddd-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564306 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564314 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564324 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f184c11a-b290-49a2-bab7-0e3bad0dbddd-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:40.564530 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:40.564337 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195e4171-ca09-4c7f-afbe-08a2fa33a736-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:41.098566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.098527 5520 generic.go:356] "Generic (PLEG): container finished" podID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerID="cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4" exitCode=0 Oct 31 03:56:41.099071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.098604 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" Oct 31 03:56:41.099071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.098615 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" event={"ID":"195e4171-ca09-4c7f-afbe-08a2fa33a736","Type":"ContainerDied","Data":"cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4"} Oct 31 03:56:41.099071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.098666 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52" event={"ID":"195e4171-ca09-4c7f-afbe-08a2fa33a736","Type":"ContainerDied","Data":"5e5d29eaa6ac787936c8f9745f6ca0691bf5c89c10fcd166f4716ed0434955c9"} Oct 31 03:56:41.099071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.098689 5520 scope.go:122] "RemoveContainer" containerID="cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4" Oct 31 03:56:41.101046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.100762 5520 generic.go:356] "Generic (PLEG): container finished" podID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerID="945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3" exitCode=0 Oct 31 03:56:41.101046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.100846 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" Oct 31 03:56:41.101297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.100846 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" event={"ID":"3b46c5ba-21f9-4d91-adbc-16e654451c06","Type":"ContainerDied","Data":"945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3"} Oct 31 03:56:41.101297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.101098 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j" event={"ID":"3b46c5ba-21f9-4d91-adbc-16e654451c06","Type":"ContainerDied","Data":"fb72c48f0c6ec2aa55754b28355fd81b132dd1cd429493dfcc7757eb635d4706"} Oct 31 03:56:41.102648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.102627 5520 generic.go:356] "Generic (PLEG): container finished" podID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerID="5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c" exitCode=0 Oct 31 03:56:41.102776 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.102754 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" Oct 31 03:56:41.102853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.102759 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" event={"ID":"f184c11a-b290-49a2-bab7-0e3bad0dbddd","Type":"ContainerDied","Data":"5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c"} Oct 31 03:56:41.102915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.102869 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" event={"ID":"f184c11a-b290-49a2-bab7-0e3bad0dbddd","Type":"ContainerDied","Data":"f45c5ab6f0ea8f52a16c73f6b36b3333dff146bc47659c1e58af91944086d876"} Oct 31 03:56:41.118376 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.118346 5520 scope.go:122] "RemoveContainer" containerID="cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4" Oct 31 03:56:41.118695 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:41.118672 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4\": container with ID starting with cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4 not found: ID does not exist" containerID="cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4" Oct 31 03:56:41.118815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.118708 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4"} err="failed to get container status \"cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4\": rpc error: code = NotFound desc = could not find container \"cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4\": container with ID starting with cee490d81e05c5aefce2a0d600b587598c2ea9c62e3c5369cdc80cc62a4b1bc4 not found: ID does not exist" Oct 31 03:56:41.118815 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.118732 5520 scope.go:122] "RemoveContainer" containerID="945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3" Oct 31 03:56:41.128843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.128760 5520 scope.go:122] "RemoveContainer" containerID="945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3" Oct 31 03:56:41.129104 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:41.129080 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3\": container with ID starting with 945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3 not found: ID does not exist" containerID="945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3" Oct 31 03:56:41.129241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.129114 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3"} err="failed to get container status \"945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3\": rpc error: code = NotFound desc = could not find container \"945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3\": container with ID starting with 945f416dbd39c25fee6c55d22c5adb80392ba597afb3f40548cc273f4df0ebd3 not found: ID does not exist" Oct 31 03:56:41.129241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.129138 5520 scope.go:122] "RemoveContainer" containerID="5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c" Oct 31 03:56:41.140407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.140374 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52"] Oct 31 03:56:41.144455 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.144431 5520 scope.go:122] "RemoveContainer" containerID="5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c" Oct 31 03:56:41.146359 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:41.146325 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c\": container with ID starting with 5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c not found: ID does not exist" containerID="5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c" Oct 31 03:56:41.146495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.146371 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c"} err="failed to get container status \"5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c\": rpc error: code = NotFound desc = could not find container \"5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c\": container with ID starting with 5e3881b900591fc0b058318ffd85073b23fe1fdd856117896f12e5b1bac59e3c not found: ID does not exist" Oct 31 03:56:41.147216 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.147175 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/premium-w1-6c6050-kserve-5b9fcb8f7c-nzx52"] Oct 31 03:56:41.158268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.157989 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j"] Oct 31 03:56:41.162505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.162479 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-2-w1-6c6050-kserve-57f8cff5-ktn5j"] Oct 31 03:56:41.167694 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:41.167668 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf184c11a_b290_49a2_bab7_0e3bad0dbddd.slice/crio-f45c5ab6f0ea8f52a16c73f6b36b3333dff146bc47659c1e58af91944086d876\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf184c11a_b290_49a2_bab7_0e3bad0dbddd.slice\": RecentStats: unable to find data in memory cache]" Oct 31 03:56:41.172748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.172719 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k"] Oct 31 03:56:41.180562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.180520 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k"] Oct 31 03:56:41.249585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.249540 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/simulator-w1-6c6050-kserve-79856b8cfb-6mf6k" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:56:41.919409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.919370 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" path="/var/lib/kubelet/pods/195e4171-ca09-4c7f-afbe-08a2fa33a736/volumes" Oct 31 03:56:41.920063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.920038 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" path="/var/lib/kubelet/pods/3b46c5ba-21f9-4d91-adbc-16e654451c06/volumes" Oct 31 03:56:41.920621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:41.920603 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" path="/var/lib/kubelet/pods/f184c11a-b290-49a2-bab7-0e3bad0dbddd/volumes" Oct 31 03:56:42.997014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:42.996959 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs"] Oct 31 03:56:42.998095 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:42.998018 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" containerID="cri-o://dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10" gracePeriod=30 Oct 31 03:56:43.379635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:43.379540 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" podUID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:56:45.707343 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:45.707306 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:45.707759 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:45.707379 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:46.291669 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:46.291627 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" podUID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:56:46.493761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:46.491594 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq"] Oct 31 03:56:46.493761 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:46.491988 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" containerID="cri-o://2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26" gracePeriod=30 Oct 31 03:56:47.398487 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.398438 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g"] Oct 31 03:56:47.400543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400504 5520 cpu_manager.go:414] "containerMap: removing container" podUID="aa02636b-059a-434e-b5a0-5d93f8313dc3" containerName="maas-api" Oct 31 03:56:47.400543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400535 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa02636b-059a-434e-b5a0-5d93f8313dc3" containerName="maas-api" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400556 5520 cpu_manager.go:414] "containerMap: removing container" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400564 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400580 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400588 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400598 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400606 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400619 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c1381bbc-d77c-42c2-8ab7-d536c4251d1c" containerName="payload-pre-processing" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400626 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1381bbc-d77c-42c2-8ab7-d536c4251d1c" containerName="payload-pre-processing" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400640 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400646 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400661 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" containerName="maas-api" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400667 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" containerName="maas-api" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400678 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400684 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400699 5520 cpu_manager.go:414] "containerMap: removing container" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400706 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400717 5520 cpu_manager.go:414] "containerMap: removing container" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400724 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400734 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400740 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400751 5520 cpu_manager.go:414] "containerMap: removing container" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400757 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400767 5520 cpu_manager.go:414] "containerMap: removing container" podUID="0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" containerName="payload-pre-processing" Oct 31 03:56:47.400760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400776 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" containerName="payload-pre-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400786 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400793 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400808 5520 cpu_manager.go:414] "containerMap: removing container" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400814 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400825 5520 cpu_manager.go:414] "containerMap: removing container" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400832 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400844 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7ed6ab6d-8790-4555-961c-9a00d02bc19a" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400851 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed6ab6d-8790-4555-961c-9a00d02bc19a" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400862 5520 cpu_manager.go:414] "containerMap: removing container" podUID="089d79dd-f7ba-478f-9bb8-701683e36f62" containerName="payload-pre-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400870 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089d79dd-f7ba-478f-9bb8-701683e36f62" containerName="payload-pre-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400882 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400891 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400901 5520 cpu_manager.go:414] "containerMap: removing container" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400907 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400918 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400924 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400951 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7e1bca48-0aeb-42b6-b774-240be3e897e9" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400958 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e1bca48-0aeb-42b6-b774-240be3e897e9" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400972 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f25e37b8-2e60-4c2a-bc75-5a9e133a8269" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400979 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25e37b8-2e60-4c2a-bc75-5a9e133a8269" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400991 5520 cpu_manager.go:414] "containerMap: removing container" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.400998 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401010 5520 cpu_manager.go:414] "containerMap: removing container" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401016 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401026 5520 cpu_manager.go:414] "containerMap: removing container" podUID="82b5e11c-6318-46e4-aa59-fb0d7b972691" containerName="maas-api" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401032 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b5e11c-6318-46e4-aa59-fb0d7b972691" containerName="maas-api" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401053 5520 cpu_manager.go:414] "containerMap: removing container" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401060 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb74b35-d0cc-4865-9617-263b41b4ec7a" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401068 5520 cpu_manager.go:414] "containerMap: removing container" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401075 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401182 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401193 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e15ae6-cdb8-46aa-b7b8-983b7c19920c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401226 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed6ab6d-8790-4555-961c-9a00d02bc19a" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401238 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="089d79dd-f7ba-478f-9bb8-701683e36f62" containerName="payload-pre-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401248 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ba2dce-da62-4cea-bfce-7c77a1c6121d" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401256 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401264 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401276 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e1bca48-0aeb-42b6-b774-240be3e897e9" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401286 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25e37b8-2e60-4c2a-bc75-5a9e133a8269" containerName="payload-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401295 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f184c11a-b290-49a2-bab7-0e3bad0dbddd" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401303 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="78562ae3-0ada-42c7-917f-52c0152b890a" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401311 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b5e11c-6318-46e4-aa59-fb0d7b972691" containerName="maas-api" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401322 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="be417875-9d21-454c-908f-d5b154479635" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401330 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa02636b-059a-434e-b5a0-5d93f8313dc3" containerName="maas-api" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401340 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="63146752-5307-488c-9217-8d41a8e28e29" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401349 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9895b9-a592-4acf-a80a-3ef016179bc2" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401357 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401365 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1381bbc-d77c-42c2-8ab7-d536c4251d1c" containerName="payload-pre-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401375 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea1f583-9a3f-47ae-a2bf-23847dfa9a54" containerName="maas-api" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401384 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5883cc3-a2f9-4082-bd89-f0940ae80531" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401394 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="195e4171-ca09-4c7f-afbe-08a2fa33a736" containerName="main" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401402 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acc8610-820f-4fc1-9fc2-760a4db6a08a" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401409 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2aec880-cd9a-46fb-99a6-4a51e8921c94" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401417 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c92f1df-97ea-4f0f-8b10-3c04f109f33c" containerName="istio-proxy" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401424 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc1a4d2-a402-470e-9dd2-31a9fb33be0d" containerName="payload-pre-processing" Oct 31 03:56:47.402038 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.401433 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b46c5ba-21f9-4d91-adbc-16e654451c06" containerName="main" Oct 31 03:56:47.407139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.407119 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.418008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.417980 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-34cb76\"/\"openshift-service-ca.crt\"" Oct 31 03:56:47.418142 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.418052 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-34cb76\"/\"kube-root-ca.crt\"" Oct 31 03:56:47.419940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.419170 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-34cb76\"/\"sub-test-model-34cb76-kserve-self-signed-certs\"" Oct 31 03:56:47.419940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.419294 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-34cb76\"/\"default-dockercfg-jp62l\"" Oct 31 03:56:47.423068 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.423045 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g"] Oct 31 03:56:47.494011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.493973 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-dshm\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.494252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.494096 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-tmp-dir\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.494252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.494156 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7460e080-d4f4-46c4-92c4-f1629f499cbb-tls-certs\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.494252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.494240 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-home\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.494451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.494273 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27h8t\" (UniqueName: \"kubernetes.io/projected/7460e080-d4f4-46c4-92c4-f1629f499cbb-kube-api-access-27h8t\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.494451 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.494309 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-model-cache\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.601147 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-tmp-dir\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.601273 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7460e080-d4f4-46c4-92c4-f1629f499cbb-tls-certs\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.601338 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-home\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.601368 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27h8t\" (UniqueName: \"kubernetes.io/projected/7460e080-d4f4-46c4-92c4-f1629f499cbb-kube-api-access-27h8t\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.601409 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-model-cache\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.601536 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-dshm\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.602345 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-home\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.602620 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-tmp-dir\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.605243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.603516 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-model-cache\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.635255 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.634518 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-dshm\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.642266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.641963 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7460e080-d4f4-46c4-92c4-f1629f499cbb-tls-certs\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.652174 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.652091 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27h8t\" (UniqueName: \"kubernetes.io/projected/7460e080-d4f4-46c4-92c4-f1629f499cbb-kube-api-access-27h8t\") pod \"sub-test-model-34cb76-kserve-94b4f9cff-wpv7g\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:47.726944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:47.726908 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:48.028477 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.028428 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g"] Oct 31 03:56:48.030467 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:56:48.030437 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7460e080_d4f4_46c4_92c4_f1629f499cbb.slice/crio-a7d7f09ed43927bc300ae93ba8806a52d70de48858701e28417bec234c5a3d58 WatchSource:0}: Error finding container a7d7f09ed43927bc300ae93ba8806a52d70de48858701e28417bec234c5a3d58: Status 404 returned error can't find the container with id a7d7f09ed43927bc300ae93ba8806a52d70de48858701e28417bec234c5a3d58 Oct 31 03:56:48.206895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.206864 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:56:48.226665 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.226630 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" event={"ID":"7460e080-d4f4-46c4-92c4-f1629f499cbb","Type":"ContainerStarted","Data":"625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76"} Oct 31 03:56:48.226844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.226680 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" event={"ID":"7460e080-d4f4-46c4-92c4-f1629f499cbb","Type":"ContainerStarted","Data":"a7d7f09ed43927bc300ae93ba8806a52d70de48858701e28417bec234c5a3d58"} Oct 31 03:56:48.228447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.228423 5520 generic.go:356] "Generic (PLEG): container finished" podID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerID="dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10" exitCode=0 Oct 31 03:56:48.228557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.228485 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" Oct 31 03:56:48.228557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.228507 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" event={"ID":"3a1d47d5-5821-40d5-8a86-ef63a1148ae6","Type":"ContainerDied","Data":"dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10"} Oct 31 03:56:48.228649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.228565 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs" event={"ID":"3a1d47d5-5821-40d5-8a86-ef63a1148ae6","Type":"ContainerDied","Data":"64c8f88005b70b3003dbbc1565cab348e600c47d290d9a060f367a90983f80c0"} Oct 31 03:56:48.228649 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.228595 5520 scope.go:122] "RemoveContainer" containerID="dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10" Oct 31 03:56:48.238359 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.238342 5520 scope.go:122] "RemoveContainer" containerID="4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1" Oct 31 03:56:48.248454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.248430 5520 scope.go:122] "RemoveContainer" containerID="dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10" Oct 31 03:56:48.249263 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:48.249241 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10\": container with ID starting with dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10 not found: ID does not exist" containerID="dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10" Oct 31 03:56:48.249421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.249392 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10"} err="failed to get container status \"dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10\": rpc error: code = NotFound desc = could not find container \"dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10\": container with ID starting with dc6622d8d9298695a359d3abe20c322c81dd141f3441a2ed5fedf0fd70da7e10 not found: ID does not exist" Oct 31 03:56:48.249551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.249530 5520 scope.go:122] "RemoveContainer" containerID="4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1" Oct 31 03:56:48.250261 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:48.249844 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1\": container with ID starting with 4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1 not found: ID does not exist" containerID="4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1" Oct 31 03:56:48.250261 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.249880 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1"} err="failed to get container status \"4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1\": rpc error: code = NotFound desc = could not find container \"4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1\": container with ID starting with 4a729ee58ff1a21e92ba9958b256555eb1884edfbe371efde41079cad380f8d1 not found: ID does not exist" Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.342764 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-credential-socket\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.342879 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-socket\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.342968 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-certs\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.343001 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-token\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-token\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.343033 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-data\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.343095 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-kube-api-access-tqnn2\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-kube-api-access-tqnn2\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.343246 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-podinfo\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343276 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.343274 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istiod-ca-cert\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.343816 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.343345 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-envoy\") pod \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\" (UID: \"3a1d47d5-5821-40d5-8a86-ef63a1148ae6\") " Oct 31 03:56:48.356331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.356280 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-data" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:48.358251 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.358177 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-socket" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:48.358454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.358413 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-credential-socket" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:48.358687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.358664 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istiod-ca-cert" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:48.360271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.359186 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-certs" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:48.361053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.361025 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-podinfo" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:48.363696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.363672 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-kube-api-access-tqnn2" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "kube-api-access-tqnn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:48.363846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.363759 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-token" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:48.364984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.364953 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-envoy" pod "3a1d47d5-5821-40d5-8a86-ef63a1148ae6" (UID: "3a1d47d5-5821-40d5-8a86-ef63a1148ae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:48.448422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.448378 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.448995 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.448972 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.449106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.449093 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.449246 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.449231 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.449358 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.449346 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.449469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.449455 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.449570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.449557 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.449655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.449644 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.449866 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.449844 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnn2\" (UniqueName: \"kubernetes.io/projected/3a1d47d5-5821-40d5-8a86-ef63a1148ae6-kube-api-access-tqnn2\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:48.552468 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.551878 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" podStartSLOduration=1.5518578459999999 podStartE2EDuration="1.551857846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:48.357823023 +0000 UTC m=+2432.552346146" watchObservedRunningTime="2026-10-31 03:56:48.551857846 +0000 UTC m=+2432.746380967" Oct 31 03:56:48.554346 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.554322 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs"] Oct 31 03:56:48.562643 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.562609 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fc5ece-openshift-default-7cfb8f46d5-8q2qs"] Oct 31 03:56:48.727311 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:48.727278 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:49.916661 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:49.916627 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" path="/var/lib/kubelet/pods/3a1d47d5-5821-40d5-8a86-ef63a1148ae6/volumes" Oct 31 03:56:50.786592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:50.786557 5520 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:50.786771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:50.786619 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:51.839745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:51.839718 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:56:51.999708 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:51.999671 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-podinfo\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.000055 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.000031 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-envoy\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.017229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.000263 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-data\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.017617 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.017593 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-certs\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.017796 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.017778 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ab80f9-6359-4b2e-bea4-2b047417170b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ab80f9-6359-4b2e-bea4-2b047417170b-istiod-ca-cert\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.018048 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.018029 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-credential-socket\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.018268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.018250 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-token\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-token\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.018489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.018473 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-socket\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.018655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.018638 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-kube-api-access-chph7\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-kube-api-access-chph7\") pod \"d1ab80f9-6359-4b2e-bea4-2b047417170b\" (UID: \"d1ab80f9-6359-4b2e-bea4-2b047417170b\") " Oct 31 03:56:52.020328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.020303 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-credential-socket" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:52.020534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.020436 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-certs" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:52.020915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.020896 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-data" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:52.021078 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.020909 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ab80f9-6359-4b2e-bea4-2b047417170b-istiod-ca-cert" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:52.021189 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.021086 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-socket" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:52.023057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.023035 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-token" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:52.027524 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.027336 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-envoy" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:52.029662 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.029329 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-kube-api-access-chph7" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "kube-api-access-chph7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:52.034390 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.034364 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-podinfo" pod "d1ab80f9-6359-4b2e-bea4-2b047417170b" (UID: "d1ab80f9-6359-4b2e-bea4-2b047417170b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121101 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121132 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chph7\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-kube-api-access-chph7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121144 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121153 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121162 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121170 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121178 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1ab80f9-6359-4b2e-bea4-2b047417170b-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121186 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1ab80f9-6359-4b2e-bea4-2b047417170b-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.121565 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.121194 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1ab80f9-6359-4b2e-bea4-2b047417170b-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.257572 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.257527 5520 generic.go:356] "Generic (PLEG): container finished" podID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerID="2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26" exitCode=0 Oct 31 03:56:52.257782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.257650 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" Oct 31 03:56:52.257782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.257678 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" event={"ID":"d1ab80f9-6359-4b2e-bea4-2b047417170b","Type":"ContainerDied","Data":"2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26"} Oct 31 03:56:52.257782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.257733 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq" event={"ID":"d1ab80f9-6359-4b2e-bea4-2b047417170b","Type":"ContainerDied","Data":"e6c02603dfee5c60f1a31190720ef3f6738336c2f8382b1dace42b727954bc76"} Oct 31 03:56:52.257782 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.257766 5520 scope.go:122] "RemoveContainer" containerID="2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26" Oct 31 03:56:52.270641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.270616 5520 scope.go:122] "RemoveContainer" containerID="2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26" Oct 31 03:56:52.271010 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:52.270988 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26\": container with ID starting with 2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26 not found: ID does not exist" containerID="2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26" Oct 31 03:56:52.271090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.271018 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26"} err="failed to get container status \"2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26\": rpc error: code = NotFound desc = could not find container \"2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26\": container with ID starting with 2c2c28ab46654f0ace6ef1e94c65d8123f3ea126cb6398ec3cd6b0bae5facb26 not found: ID does not exist" Oct 31 03:56:52.286633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.286539 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq"] Oct 31 03:56:52.289674 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:52.289646 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-7fshq"] Oct 31 03:56:53.379071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.379032 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" podUID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:56:53.535470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.525057 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p"] Oct 31 03:56:53.535470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.525396 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" containerID="cri-o://9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8" gracePeriod=30 Oct 31 03:56:53.764226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.763283 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh"] Oct 31 03:56:53.831172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.831136 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj"] Oct 31 03:56:53.831763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.831709 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" containerID="cri-o://201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97" gracePeriod=30 Oct 31 03:56:53.838313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.838281 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6"] Oct 31 03:56:53.924842 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:53.924809 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" path="/var/lib/kubelet/pods/d1ab80f9-6359-4b2e-bea4-2b047417170b/volumes" Oct 31 03:56:54.316263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.305821 5520 generic.go:356] "Generic (PLEG): container finished" podID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" containerID="ff5f99dd04f716ea631432f027e0f47347cbe1cd1d6e0cae09ce79622afa0c8a" exitCode=0 Oct 31 03:56:54.316263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.305977 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" event={"ID":"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6","Type":"ContainerDied","Data":"ff5f99dd04f716ea631432f027e0f47347cbe1cd1d6e0cae09ce79622afa0c8a"} Oct 31 03:56:54.326272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.320279 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:56:54.489893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.489854 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-plugins-config-volume\") pod \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " Oct 31 03:56:54.490409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.489950 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-kube-api-access-8h7mx\" (UniqueName: \"kubernetes.io/projected/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-kube-api-access-8h7mx\") pod \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\" (UID: \"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6\") " Oct 31 03:56:54.490409 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.490340 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-plugins-config-volume" pod "d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" (UID: "d1d83e2c-b1cb-429b-9f59-8ae21a5949c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:54.492640 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.492609 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-kube-api-access-8h7mx" pod "d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" (UID: "d1d83e2c-b1cb-429b-9f59-8ae21a5949c6"). InnerVolumeSpecName "kube-api-access-8h7mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:54.592213 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.592093 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:54.592213 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.592134 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7mx\" (UniqueName: \"kubernetes.io/projected/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6-kube-api-access-8h7mx\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:54.639153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.639095 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x"] Oct 31 03:56:54.639457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:54.639428 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" podUID="4b474775-af3d-4761-ae90-066a630dbdc8" containerName="maas-api" containerID="cri-o://e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b" gracePeriod=30 Oct 31 03:56:55.047496 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.047470 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:56:55.206810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.206605 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b474775-af3d-4761-ae90-066a630dbdc8-kube-api-access-jl25c\" (UniqueName: \"kubernetes.io/projected/4b474775-af3d-4761-ae90-066a630dbdc8-kube-api-access-jl25c\") pod \"4b474775-af3d-4761-ae90-066a630dbdc8\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " Oct 31 03:56:55.206810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.206670 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-metrics-certs\") pod \"4b474775-af3d-4761-ae90-066a630dbdc8\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " Oct 31 03:56:55.206810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.206745 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-maas-api-tls\") pod \"4b474775-af3d-4761-ae90-066a630dbdc8\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " Oct 31 03:56:55.206810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.206772 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b474775-af3d-4761-ae90-066a630dbdc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b474775-af3d-4761-ae90-066a630dbdc8-openshift-service-ca\") pod \"4b474775-af3d-4761-ae90-066a630dbdc8\" (UID: \"4b474775-af3d-4761-ae90-066a630dbdc8\") " Oct 31 03:56:55.207257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.207218 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b474775-af3d-4761-ae90-066a630dbdc8-openshift-service-ca" pod "4b474775-af3d-4761-ae90-066a630dbdc8" (UID: "4b474775-af3d-4761-ae90-066a630dbdc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:55.207369 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.207348 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b474775-af3d-4761-ae90-066a630dbdc8-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:55.209849 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.209816 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-maas-api-tls" pod "4b474775-af3d-4761-ae90-066a630dbdc8" (UID: "4b474775-af3d-4761-ae90-066a630dbdc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:55.209996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.209939 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-metrics-certs" pod "4b474775-af3d-4761-ae90-066a630dbdc8" (UID: "4b474775-af3d-4761-ae90-066a630dbdc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:55.209996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.209958 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b474775-af3d-4761-ae90-066a630dbdc8-kube-api-access-jl25c" pod "4b474775-af3d-4761-ae90-066a630dbdc8" (UID: "4b474775-af3d-4761-ae90-066a630dbdc8"). InnerVolumeSpecName "kube-api-access-jl25c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:55.308221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.308084 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl25c\" (UniqueName: \"kubernetes.io/projected/4b474775-af3d-4761-ae90-066a630dbdc8-kube-api-access-jl25c\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:55.308221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.308143 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:55.308221 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.308165 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b474775-af3d-4761-ae90-066a630dbdc8-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:55.316136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.316102 5520 generic.go:356] "Generic (PLEG): container finished" podID="4b474775-af3d-4761-ae90-066a630dbdc8" containerID="e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b" exitCode=0 Oct 31 03:56:55.316136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.316137 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" Oct 31 03:56:55.316422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.316184 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" event={"ID":"4b474775-af3d-4761-ae90-066a630dbdc8","Type":"ContainerDied","Data":"e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b"} Oct 31 03:56:55.316422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.316266 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x" event={"ID":"4b474775-af3d-4761-ae90-066a630dbdc8","Type":"ContainerDied","Data":"80f33380a30c44761d0fb4597727464b6058346cbfec8a021ed68cd533b20a70"} Oct 31 03:56:55.316422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.316297 5520 scope.go:122] "RemoveContainer" containerID="e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b" Oct 31 03:56:55.318807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.318787 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" event={"ID":"d1d83e2c-b1cb-429b-9f59-8ae21a5949c6","Type":"ContainerDied","Data":"49a8309e8aa4731624aba55dda40184c242ca73b9ad83cc02b6703e609675460"} Oct 31 03:56:55.318946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.318822 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt" Oct 31 03:56:55.330725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.330698 5520 scope.go:122] "RemoveContainer" containerID="e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b" Oct 31 03:56:55.331069 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:55.331045 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b\": container with ID starting with e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b not found: ID does not exist" containerID="e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b" Oct 31 03:56:55.331194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.331074 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b"} err="failed to get container status \"e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b\": rpc error: code = NotFound desc = could not find container \"e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b\": container with ID starting with e99e4a9c9f0ba339d23d145587744f7bf8632d0314aa3b8c487b1b2c98aa0c4b not found: ID does not exist" Oct 31 03:56:55.331194 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.331118 5520 scope.go:122] "RemoveContainer" containerID="ff5f99dd04f716ea631432f027e0f47347cbe1cd1d6e0cae09ce79622afa0c8a" Oct 31 03:56:55.348418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.348393 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x"] Oct 31 03:56:55.351421 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.351358 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-6c6050-67d6c5d668-mxj4x"] Oct 31 03:56:55.363831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.363796 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt"] Oct 31 03:56:55.367053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.367028 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-86a40d-5bcffb7d8f-j24dt"] Oct 31 03:56:55.652252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.645156 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd"] Oct 31 03:56:55.652252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.646138 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" podUID="462c66e6-49cc-4676-b108-e10daa5b4fc5" containerName="maas-api" containerID="cri-o://605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3" gracePeriod=30 Oct 31 03:56:55.771687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.771387 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:56:55.940283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.935002 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-plugins-config-volume\") pod \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " Oct 31 03:56:55.940283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.935179 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-kube-api-access-wkldc\" (UniqueName: \"kubernetes.io/projected/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-kube-api-access-wkldc\") pod \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\" (UID: \"b4bed3b9-0e41-42dd-a0af-86a9526f3b82\") " Oct 31 03:56:55.940283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.938717 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-kube-api-access-wkldc" pod "b4bed3b9-0e41-42dd-a0af-86a9526f3b82" (UID: "b4bed3b9-0e41-42dd-a0af-86a9526f3b82"). InnerVolumeSpecName "kube-api-access-wkldc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:55.940283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.939060 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-plugins-config-volume" pod "b4bed3b9-0e41-42dd-a0af-86a9526f3b82" (UID: "b4bed3b9-0e41-42dd-a0af-86a9526f3b82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:55.959687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.959653 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b474775-af3d-4761-ae90-066a630dbdc8" path="/var/lib/kubelet/pods/4b474775-af3d-4761-ae90-066a630dbdc8/volumes" Oct 31 03:56:55.974224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:55.961287 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" path="/var/lib/kubelet/pods/d1d83e2c-b1cb-429b-9f59-8ae21a5949c6/volumes" Oct 31 03:56:56.037619 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.037583 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.037802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.037622 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkldc\" (UniqueName: \"kubernetes.io/projected/b4bed3b9-0e41-42dd-a0af-86a9526f3b82-kube-api-access-wkldc\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.109231 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.108138 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:56:56.240176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.240041 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462c66e6-49cc-4676-b108-e10daa5b4fc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c66e6-49cc-4676-b108-e10daa5b4fc5-openshift-service-ca\") pod \"462c66e6-49cc-4676-b108-e10daa5b4fc5\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " Oct 31 03:56:56.240176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.240130 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462c66e6-49cc-4676-b108-e10daa5b4fc5-kube-api-access-nxc66\" (UniqueName: \"kubernetes.io/projected/462c66e6-49cc-4676-b108-e10daa5b4fc5-kube-api-access-nxc66\") pod \"462c66e6-49cc-4676-b108-e10daa5b4fc5\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " Oct 31 03:56:56.240442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.240217 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-maas-api-tls\") pod \"462c66e6-49cc-4676-b108-e10daa5b4fc5\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " Oct 31 03:56:56.240442 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.240314 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-metrics-certs\") pod \"462c66e6-49cc-4676-b108-e10daa5b4fc5\" (UID: \"462c66e6-49cc-4676-b108-e10daa5b4fc5\") " Oct 31 03:56:56.244378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.241286 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462c66e6-49cc-4676-b108-e10daa5b4fc5-openshift-service-ca" pod "462c66e6-49cc-4676-b108-e10daa5b4fc5" (UID: "462c66e6-49cc-4676-b108-e10daa5b4fc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:56.244378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.243342 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-metrics-certs" pod "462c66e6-49cc-4676-b108-e10daa5b4fc5" (UID: "462c66e6-49cc-4676-b108-e10daa5b4fc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:56.248939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.246706 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462c66e6-49cc-4676-b108-e10daa5b4fc5-kube-api-access-nxc66" pod "462c66e6-49cc-4676-b108-e10daa5b4fc5" (UID: "462c66e6-49cc-4676-b108-e10daa5b4fc5"). InnerVolumeSpecName "kube-api-access-nxc66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:56.249220 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.249174 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-maas-api-tls" pod "462c66e6-49cc-4676-b108-e10daa5b4fc5" (UID: "462c66e6-49cc-4676-b108-e10daa5b4fc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:56.326370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.326332 5520 generic.go:356] "Generic (PLEG): container finished" podID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" containerID="578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12" exitCode=0 Oct 31 03:56:56.326570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.326511 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" Oct 31 03:56:56.327028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.327005 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" event={"ID":"b4bed3b9-0e41-42dd-a0af-86a9526f3b82","Type":"ContainerDied","Data":"578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12"} Oct 31 03:56:56.327153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.327096 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf" event={"ID":"b4bed3b9-0e41-42dd-a0af-86a9526f3b82","Type":"ContainerDied","Data":"92622aad593d07da663e68b45a6878d771279a37b11ee19c5ba62e2e2a6125e4"} Oct 31 03:56:56.327153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.327129 5520 scope.go:122] "RemoveContainer" containerID="578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12" Oct 31 03:56:56.330465 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.330338 5520 generic.go:356] "Generic (PLEG): container finished" podID="462c66e6-49cc-4676-b108-e10daa5b4fc5" containerID="605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3" exitCode=0 Oct 31 03:56:56.330586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.330491 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" event={"ID":"462c66e6-49cc-4676-b108-e10daa5b4fc5","Type":"ContainerDied","Data":"605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3"} Oct 31 03:56:56.330586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.330536 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" event={"ID":"462c66e6-49cc-4676-b108-e10daa5b4fc5","Type":"ContainerDied","Data":"5de86b185de6ad84d7913fb696a3f4c7cbb6ab186f9909e87fa50516dc7d11f7"} Oct 31 03:56:56.330705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.330622 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.341636 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462c66e6-49cc-4676-b108-e10daa5b4fc5-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.341663 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxc66\" (UniqueName: \"kubernetes.io/projected/462c66e6-49cc-4676-b108-e10daa5b4fc5-kube-api-access-nxc66\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.341679 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.341693 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/462c66e6-49cc-4676-b108-e10daa5b4fc5-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.344511 5520 scope.go:122] "RemoveContainer" containerID="578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:56.344873 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12\": container with ID starting with 578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12 not found: ID does not exist" containerID="578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.344906 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12"} err="failed to get container status \"578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12\": rpc error: code = NotFound desc = could not find container \"578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12\": container with ID starting with 578457df90cb13b201c6fb9f81f9b66c2d8b4ad34ba72f4d17e5e1b6dcf48f12 not found: ID does not exist" Oct 31 03:56:56.346612 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.344927 5520 scope.go:122] "RemoveContainer" containerID="605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3" Oct 31 03:56:56.354028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.354000 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf"] Oct 31 03:56:56.356952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.356925 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6c6050-6cb45c4cb9-vr4sf"] Oct 31 03:56:56.357719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.357699 5520 scope.go:122] "RemoveContainer" containerID="605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3" Oct 31 03:56:56.358087 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:56.358065 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3\": container with ID starting with 605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3 not found: ID does not exist" containerID="605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3" Oct 31 03:56:56.358166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.358094 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3"} err="failed to get container status \"605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3\": rpc error: code = NotFound desc = could not find container \"605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3\": container with ID starting with 605c43ccfd54ff70080497e96c19dc5921730c8f07c7559f1c4b40077405edc3 not found: ID does not exist" Oct 31 03:56:56.376507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.372135 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd"] Oct 31 03:56:56.376507 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:56.376276 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5396c2-58f77c569-679nd"] Oct 31 03:56:57.728121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:57.728084 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:57.747074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:57.747044 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:57.920020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:57.919985 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462c66e6-49cc-4676-b108-e10daa5b4fc5" path="/var/lib/kubelet/pods/462c66e6-49cc-4676-b108-e10daa5b4fc5/volumes" Oct 31 03:56:57.920551 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:57.920528 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" path="/var/lib/kubelet/pods/b4bed3b9-0e41-42dd-a0af-86a9526f3b82/volumes" Oct 31 03:56:58.363336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.363307 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:56:58.608226 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:58.606304 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8b627d_d57a_4bfe_ad48_15ca5bfc77c9.slice/crio-conmon-9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:56:58.608226 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:58.607461 5520 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8b627d_d57a_4bfe_ad48_15ca5bfc77c9.slice/crio-conmon-9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:56:58.764069 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.764026 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" podUID="6388c710-6b06-4d75-a1c8-945d2c462dd3" containerName="payload-processing" containerID="cri-o://bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c" gracePeriod=25 Oct 31 03:56:58.839527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.839472 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" podUID="6ba243e9-16f1-45ff-9133-924a298d0fe6" containerName="payload-pre-processing" containerID="cri-o://b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec" gracePeriod=25 Oct 31 03:56:58.952677 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.952644 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:56:58.975742 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.975706 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-socket\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.975922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.975759 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-token\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-token\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.975922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.975832 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-data\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.975922 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.975870 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istiod-ca-cert\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.976082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.975921 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-kube-api-access-gnqjz\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-kube-api-access-gnqjz\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.976082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.976026 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-credential-socket\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.976082 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.976071 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-envoy\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.976277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.976119 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-certs\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.976277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.976185 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-podinfo\") pod \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\" (UID: \"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9\") " Oct 31 03:56:58.989978 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.989938 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-socket" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:58.993821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.993294 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-data" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:58.993821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.993308 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-credential-socket" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:58.993821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.993636 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istiod-ca-cert" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:58.993821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.993744 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-certs" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:58.994706 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.994682 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-token" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:58.995909 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.995877 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-envoy" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:58.997903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.997871 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-kube-api-access-gnqjz" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "kube-api-access-gnqjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:58.998417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:58.998392 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-podinfo" pod "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" (UID: "bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078707 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078758 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078773 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078786 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078801 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnqjz\" (UniqueName: \"kubernetes.io/projected/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-kube-api-access-gnqjz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078833 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078846 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078859 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.080800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.078872 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.140732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.140692 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:56:59.180046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.180014 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba243e9-16f1-45ff-9133-924a298d0fe6-kube-api-access-5p5pl\" (UniqueName: \"kubernetes.io/projected/6ba243e9-16f1-45ff-9133-924a298d0fe6-kube-api-access-5p5pl\") pod \"6ba243e9-16f1-45ff-9133-924a298d0fe6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " Oct 31 03:56:59.180242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.180161 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba243e9-16f1-45ff-9133-924a298d0fe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba243e9-16f1-45ff-9133-924a298d0fe6-plugins-config-volume\") pod \"6ba243e9-16f1-45ff-9133-924a298d0fe6\" (UID: \"6ba243e9-16f1-45ff-9133-924a298d0fe6\") " Oct 31 03:56:59.181171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.181111 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba243e9-16f1-45ff-9133-924a298d0fe6-plugins-config-volume" pod "6ba243e9-16f1-45ff-9133-924a298d0fe6" (UID: "6ba243e9-16f1-45ff-9133-924a298d0fe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:59.184022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.183989 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba243e9-16f1-45ff-9133-924a298d0fe6-kube-api-access-5p5pl" pod "6ba243e9-16f1-45ff-9133-924a298d0fe6" (UID: "6ba243e9-16f1-45ff-9133-924a298d0fe6"). InnerVolumeSpecName "kube-api-access-5p5pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:59.185187 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.185165 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:56:59.216349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.216006 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:56:59.288608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288570 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-credential-socket\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.288812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288656 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6388c710-6b06-4d75-a1c8-945d2c462dd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6388c710-6b06-4d75-a1c8-945d2c462dd3-plugins-config-volume\") pod \"6388c710-6b06-4d75-a1c8-945d2c462dd3\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " Oct 31 03:56:59.288812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288688 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-token\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-token\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.288812 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288723 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-podinfo\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.289005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288821 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-certs\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.289005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288864 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-data\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.289005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288921 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa1d260-f10c-4faf-9a19-119d95aa59a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa1d260-f10c-4faf-9a19-119d95aa59a6-istiod-ca-cert\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.289005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288903 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-credential-socket" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:59.289005 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.288953 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6388c710-6b06-4d75-a1c8-945d2c462dd3-kube-api-access-bsnj7\" (UniqueName: \"kubernetes.io/projected/6388c710-6b06-4d75-a1c8-945d2c462dd3-kube-api-access-bsnj7\") pod \"6388c710-6b06-4d75-a1c8-945d2c462dd3\" (UID: \"6388c710-6b06-4d75-a1c8-945d2c462dd3\") " Oct 31 03:56:59.289310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.289009 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-kube-api-access-6q6vp\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-kube-api-access-6q6vp\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.289310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.289054 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-socket\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.289310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.289122 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-envoy\") pod \"caa1d260-f10c-4faf-9a19-119d95aa59a6\" (UID: \"caa1d260-f10c-4faf-9a19-119d95aa59a6\") " Oct 31 03:56:59.289310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.289193 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6388c710-6b06-4d75-a1c8-945d2c462dd3-plugins-config-volume" pod "6388c710-6b06-4d75-a1c8-945d2c462dd3" (UID: "6388c710-6b06-4d75-a1c8-945d2c462dd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:59.289532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.289483 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-certs" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:59.289532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.289493 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-socket" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.289940 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa1d260-f10c-4faf-9a19-119d95aa59a6-istiod-ca-cert" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290073 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p5pl\" (UniqueName: \"kubernetes.io/projected/6ba243e9-16f1-45ff-9133-924a298d0fe6-kube-api-access-5p5pl\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290095 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290111 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caa1d260-f10c-4faf-9a19-119d95aa59a6-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290126 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba243e9-16f1-45ff-9133-924a298d0fe6-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290143 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290158 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290173 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6388c710-6b06-4d75-a1c8-945d2c462dd3-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.290533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.290500 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-data" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:59.291652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.291615 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-podinfo" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:59.292832 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.292793 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-kube-api-access-6q6vp" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "kube-api-access-6q6vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:59.293067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.293034 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-envoy" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:59.293154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.293124 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6388c710-6b06-4d75-a1c8-945d2c462dd3-kube-api-access-bsnj7" pod "6388c710-6b06-4d75-a1c8-945d2c462dd3" (UID: "6388c710-6b06-4d75-a1c8-945d2c462dd3"). InnerVolumeSpecName "kube-api-access-bsnj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:59.294067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.294042 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-token" pod "caa1d260-f10c-4faf-9a19-119d95aa59a6" (UID: "caa1d260-f10c-4faf-9a19-119d95aa59a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:59.361620 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.361581 5520 generic.go:356] "Generic (PLEG): container finished" podID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerID="201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97" exitCode=0 Oct 31 03:56:59.361822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.361631 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" Oct 31 03:56:59.361822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.361662 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" event={"ID":"caa1d260-f10c-4faf-9a19-119d95aa59a6","Type":"ContainerDied","Data":"201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97"} Oct 31 03:56:59.361822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.361717 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj" event={"ID":"caa1d260-f10c-4faf-9a19-119d95aa59a6","Type":"ContainerDied","Data":"c0093b38b79d9ca1e70d95a83f13d68cb2e0646eadb1ed063f91683b94f2d81f"} Oct 31 03:56:59.361822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.361734 5520 scope.go:122] "RemoveContainer" containerID="201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97" Oct 31 03:56:59.367239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.363766 5520 generic.go:356] "Generic (PLEG): container finished" podID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerID="9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8" exitCode=0 Oct 31 03:56:59.367239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.363824 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" event={"ID":"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9","Type":"ContainerDied","Data":"9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8"} Oct 31 03:56:59.367239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.363892 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" event={"ID":"bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9","Type":"ContainerDied","Data":"126e34995ffd1d91a7fa179a133108579647d6ba7c51e233cf77e5e77b7875f1"} Oct 31 03:56:59.367239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.363834 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p" Oct 31 03:56:59.370333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.369613 5520 generic.go:356] "Generic (PLEG): container finished" podID="6388c710-6b06-4d75-a1c8-945d2c462dd3" containerID="bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c" exitCode=0 Oct 31 03:56:59.370333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.369811 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" event={"ID":"6388c710-6b06-4d75-a1c8-945d2c462dd3","Type":"ContainerDied","Data":"bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c"} Oct 31 03:56:59.370333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.369853 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" event={"ID":"6388c710-6b06-4d75-a1c8-945d2c462dd3","Type":"ContainerDied","Data":"13bf1075e52a6070a918f5810a46364c097c5d86c062a9020383a008d2cd1abc"} Oct 31 03:56:59.370333 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.369950 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh" Oct 31 03:56:59.371587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.371563 5520 generic.go:356] "Generic (PLEG): container finished" podID="6ba243e9-16f1-45ff-9133-924a298d0fe6" containerID="b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec" exitCode=0 Oct 31 03:56:59.372301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.372279 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" Oct 31 03:56:59.373008 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.372964 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" event={"ID":"6ba243e9-16f1-45ff-9133-924a298d0fe6","Type":"ContainerDied","Data":"b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec"} Oct 31 03:56:59.373170 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.373008 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6" event={"ID":"6ba243e9-16f1-45ff-9133-924a298d0fe6","Type":"ContainerDied","Data":"7e9873b2986e576420fb88f96967b7aac39741cfc2dfbf507601117410eac50a"} Oct 31 03:56:59.387403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.382065 5520 scope.go:122] "RemoveContainer" containerID="201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97" Oct 31 03:56:59.387403 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:59.382889 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97\": container with ID starting with 201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97 not found: ID does not exist" containerID="201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97" Oct 31 03:56:59.387403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.383007 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97"} err="failed to get container status \"201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97\": rpc error: code = NotFound desc = could not find container \"201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97\": container with ID starting with 201f5dce022229f16908b53f57c2e8fb2ab3e0395197f2d2ab215033c357ba97 not found: ID does not exist" Oct 31 03:56:59.387403 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.383260 5520 scope.go:122] "RemoveContainer" containerID="9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8" Oct 31 03:56:59.392234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.392190 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.392340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.392267 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.392340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.392290 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.392340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.392304 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caa1d260-f10c-4faf-9a19-119d95aa59a6-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.392517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.392342 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnj7\" (UniqueName: \"kubernetes.io/projected/6388c710-6b06-4d75-a1c8-945d2c462dd3-kube-api-access-bsnj7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.392517 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.392380 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6vp\" (UniqueName: \"kubernetes.io/projected/caa1d260-f10c-4faf-9a19-119d95aa59a6-kube-api-access-6q6vp\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:59.411385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.411356 5520 scope.go:122] "RemoveContainer" containerID="9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8" Oct 31 03:56:59.412659 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:59.411911 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8\": container with ID starting with 9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8 not found: ID does not exist" containerID="9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8" Oct 31 03:56:59.412659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.411955 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8"} err="failed to get container status \"9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8\": rpc error: code = NotFound desc = could not find container \"9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8\": container with ID starting with 9d58e053f03a2b19d48ade1acee3ebeab397e7e9e3a1f6aee864d85f2009aab8 not found: ID does not exist" Oct 31 03:56:59.412659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.411977 5520 scope.go:122] "RemoveContainer" containerID="bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c" Oct 31 03:56:59.413303 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.413272 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj"] Oct 31 03:56:59.415540 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.415522 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-d7ngj"] Oct 31 03:56:59.423914 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.423887 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p"] Oct 31 03:56:59.425252 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.425230 5520 scope.go:122] "RemoveContainer" containerID="bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c" Oct 31 03:56:59.425592 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:59.425541 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c\": container with ID starting with bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c not found: ID does not exist" containerID="bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c" Oct 31 03:56:59.425592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.425574 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c"} err="failed to get container status \"bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c\": rpc error: code = NotFound desc = could not find container \"bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c\": container with ID starting with bbae671d26ab904475eb8a9eed42febfecc9878ff4b78d4f4db5c63f930e054c not found: ID does not exist" Oct 31 03:56:59.425767 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.425599 5520 scope.go:122] "RemoveContainer" containerID="b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec" Oct 31 03:56:59.430994 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.430972 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-mpv7p"] Oct 31 03:56:59.435695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.435675 5520 scope.go:122] "RemoveContainer" containerID="b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec" Oct 31 03:56:59.436026 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:56:59.435998 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec\": container with ID starting with b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec not found: ID does not exist" containerID="b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec" Oct 31 03:56:59.436089 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.436030 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec"} err="failed to get container status \"b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec\": rpc error: code = NotFound desc = could not find container \"b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec\": container with ID starting with b9a77bb1897fb782382332582fc6629cee3e412a7823a40546da48f3773a01ec not found: ID does not exist" Oct 31 03:56:59.440417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.440380 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh"] Oct 31 03:56:59.443892 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.443872 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5396c2-6d7f6cc687-sw6lh"] Oct 31 03:56:59.455527 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.455502 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6"] Oct 31 03:56:59.458386 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.458364 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5396c2-7545849b99-mvdq6"] Oct 31 03:56:59.917679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.917644 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6388c710-6b06-4d75-a1c8-945d2c462dd3" path="/var/lib/kubelet/pods/6388c710-6b06-4d75-a1c8-945d2c462dd3/volumes" Oct 31 03:56:59.918103 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.918085 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba243e9-16f1-45ff-9133-924a298d0fe6" path="/var/lib/kubelet/pods/6ba243e9-16f1-45ff-9133-924a298d0fe6/volumes" Oct 31 03:56:59.918696 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.918675 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" path="/var/lib/kubelet/pods/bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9/volumes" Oct 31 03:56:59.919491 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:56:59.919477 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" path="/var/lib/kubelet/pods/caa1d260-f10c-4faf-9a19-119d95aa59a6/volumes" Oct 31 03:57:06.480057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480019 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf"] Oct 31 03:57:06.480483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480456 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" Oct 31 03:57:06.480483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480469 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" Oct 31 03:57:06.480483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480478 5520 cpu_manager.go:414] "containerMap: removing container" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" Oct 31 03:57:06.480483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480483 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480493 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480498 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480511 5520 cpu_manager.go:414] "containerMap: removing container" podUID="6388c710-6b06-4d75-a1c8-945d2c462dd3" containerName="payload-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480516 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6388c710-6b06-4d75-a1c8-945d2c462dd3" containerName="payload-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480527 5520 cpu_manager.go:414] "containerMap: removing container" podUID="6ba243e9-16f1-45ff-9133-924a298d0fe6" containerName="payload-pre-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480533 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba243e9-16f1-45ff-9133-924a298d0fe6" containerName="payload-pre-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480541 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480546 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480563 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" containerName="payload-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480568 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" containerName="payload-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480578 5520 cpu_manager.go:414] "containerMap: removing container" podUID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" containerName="payload-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480582 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" containerName="payload-processing" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480591 5520 cpu_manager.go:414] "containerMap: removing container" podUID="4b474775-af3d-4761-ae90-066a630dbdc8" containerName="maas-api" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480595 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b474775-af3d-4761-ae90-066a630dbdc8" containerName="maas-api" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480602 5520 cpu_manager.go:414] "containerMap: removing container" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480608 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480620 5520 cpu_manager.go:414] "containerMap: removing container" podUID="462c66e6-49cc-4676-b108-e10daa5b4fc5" containerName="maas-api" Oct 31 03:57:06.480622 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480628 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462c66e6-49cc-4676-b108-e10daa5b4fc5" containerName="maas-api" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480701 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba243e9-16f1-45ff-9133-924a298d0fe6" containerName="payload-pre-processing" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480714 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480723 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bed3b9-0e41-42dd-a0af-86a9526f3b82" containerName="payload-processing" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480731 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d83e2c-b1cb-429b-9f59-8ae21a5949c6" containerName="payload-processing" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480739 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b474775-af3d-4761-ae90-066a630dbdc8" containerName="maas-api" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480745 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa1d260-f10c-4faf-9a19-119d95aa59a6" containerName="istio-proxy" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480751 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="462c66e6-49cc-4676-b108-e10daa5b4fc5" containerName="maas-api" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480757 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ab80f9-6359-4b2e-bea4-2b047417170b" containerName="istio-proxy" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480762 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5efbc60-6ff8-43eb-bfa4-62687f95d1bf" containerName="istio-proxy" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480769 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8b627d-d57a-4bfe-ad48-15ca5bfc77c9" containerName="istio-proxy" Oct 31 03:57:06.481304 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.480776 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="6388c710-6b06-4d75-a1c8-945d2c462dd3" containerName="payload-processing" Oct 31 03:57:06.485171 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.485153 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.492493 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.492466 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2"] Oct 31 03:57:06.493298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.493273 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1d47d5-5821-40d5-8a86-ef63a1148ae6" containerName="istio-proxy" Oct 31 03:57:06.496906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.496880 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.497042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.496936 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf"] Oct 31 03:57:06.513942 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.513907 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2"] Oct 31 03:57:06.555314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.555281 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:57:06.555481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.555358 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:57:06.588052 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.588018 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.588265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.588062 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rs2f\" (UniqueName: \"kubernetes.io/projected/bdea8669-621e-4cae-8b36-d6057ba7955d-kube-api-access-9rs2f\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.588351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.588256 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.588351 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.588302 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.588470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.588400 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.588470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.588460 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffb2l\" (UniqueName: \"kubernetes.io/projected/180074c2-4d96-4581-a301-a7e0e3b9ec27-kube-api-access-ffb2l\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.689579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.689545 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.689579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.689592 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.689837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.689652 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.689837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.689688 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffb2l\" (UniqueName: \"kubernetes.io/projected/180074c2-4d96-4581-a301-a7e0e3b9ec27-kube-api-access-ffb2l\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.689837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.689798 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.690054 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.689868 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rs2f\" (UniqueName: \"kubernetes.io/projected/bdea8669-621e-4cae-8b36-d6057ba7955d-kube-api-access-9rs2f\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.690406 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.690382 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.690666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.690643 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.692154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.692133 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.692436 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.692416 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.704003 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.703973 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffb2l\" (UniqueName: \"kubernetes.io/projected/180074c2-4d96-4581-a301-a7e0e3b9ec27-kube-api-access-ffb2l\") pod \"maas-api-revoke-sub-d859f302-lxdp2\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:06.704373 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.704354 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rs2f\" (UniqueName: \"kubernetes.io/projected/bdea8669-621e-4cae-8b36-d6057ba7955d-kube-api-access-9rs2f\") pod \"maas-api-revoke-sub-a9c5211f-qwvkf\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.803191 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.803101 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:06.818794 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:06.818762 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:07.093169 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.093140 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf"] Oct 31 03:57:07.095182 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:57:07.095146 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdea8669_621e_4cae_8b36_d6057ba7955d.slice/crio-2bcf7329e1c8daa3005576dc49b5e647dea366f7a3feb9bbe15bf0c918d857ab WatchSource:0}: Error finding container 2bcf7329e1c8daa3005576dc49b5e647dea366f7a3feb9bbe15bf0c918d857ab: Status 404 returned error can't find the container with id 2bcf7329e1c8daa3005576dc49b5e647dea366f7a3feb9bbe15bf0c918d857ab Oct 31 03:57:07.114153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.114031 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2"] Oct 31 03:57:07.117040 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:57:07.117012 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180074c2_4d96_4581_a301_a7e0e3b9ec27.slice/crio-73c6ede79ba5bebb0b43f0ff7563951c904078c14f20ea158413baf9c92abdc0 WatchSource:0}: Error finding container 73c6ede79ba5bebb0b43f0ff7563951c904078c14f20ea158413baf9c92abdc0: Status 404 returned error can't find the container with id 73c6ede79ba5bebb0b43f0ff7563951c904078c14f20ea158413baf9c92abdc0 Oct 31 03:57:07.445223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.445153 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" event={"ID":"180074c2-4d96-4581-a301-a7e0e3b9ec27","Type":"ContainerStarted","Data":"ed75e0299664dc04f3f15cc52322647cacaadc1da771e0d50320506f93084cef"} Oct 31 03:57:07.445411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.445241 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" event={"ID":"180074c2-4d96-4581-a301-a7e0e3b9ec27","Type":"ContainerStarted","Data":"73c6ede79ba5bebb0b43f0ff7563951c904078c14f20ea158413baf9c92abdc0"} Oct 31 03:57:07.447124 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.447092 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" event={"ID":"bdea8669-621e-4cae-8b36-d6057ba7955d","Type":"ContainerStarted","Data":"160800bd78bd5ac7549117d25b9e54e57dbca0f68ca1a7e7f8bc638f2a1d99f5"} Oct 31 03:57:07.447264 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.447151 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" event={"ID":"bdea8669-621e-4cae-8b36-d6057ba7955d","Type":"ContainerStarted","Data":"2bcf7329e1c8daa3005576dc49b5e647dea366f7a3feb9bbe15bf0c918d857ab"} Oct 31 03:57:07.463971 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.463921 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" podStartSLOduration=1.4639073040000001 podStartE2EDuration="1.463907304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:07.461028049 +0000 UTC m=+2451.655551199" watchObservedRunningTime="2026-10-31 03:57:07.463907304 +0000 UTC m=+2451.658430421" Oct 31 03:57:07.483679 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:07.483605 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" podStartSLOduration=1.483589791 podStartE2EDuration="1.483589791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:07.481110401 +0000 UTC m=+2451.675633581" watchObservedRunningTime="2026-10-31 03:57:07.483589791 +0000 UTC m=+2451.678112911" Oct 31 03:57:08.456960 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:08.456921 5520 generic.go:356] "Generic (PLEG): container finished" podID="180074c2-4d96-4581-a301-a7e0e3b9ec27" containerID="ed75e0299664dc04f3f15cc52322647cacaadc1da771e0d50320506f93084cef" exitCode=0 Oct 31 03:57:08.456960 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:08.456948 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" event={"ID":"180074c2-4d96-4581-a301-a7e0e3b9ec27","Type":"ContainerDied","Data":"ed75e0299664dc04f3f15cc52322647cacaadc1da771e0d50320506f93084cef"} Oct 31 03:57:08.459071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:08.459045 5520 generic.go:356] "Generic (PLEG): container finished" podID="bdea8669-621e-4cae-8b36-d6057ba7955d" containerID="160800bd78bd5ac7549117d25b9e54e57dbca0f68ca1a7e7f8bc638f2a1d99f5" exitCode=0 Oct 31 03:57:08.459242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:08.459101 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" event={"ID":"bdea8669-621e-4cae-8b36-d6057ba7955d","Type":"ContainerDied","Data":"160800bd78bd5ac7549117d25b9e54e57dbca0f68ca1a7e7f8bc638f2a1d99f5"} Oct 31 03:57:09.373746 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:57:09.373703 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:57:09.662368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.662339 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:09.665532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.665508 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:09.742786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.742752 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdea8669-621e-4cae-8b36-d6057ba7955d-kube-api-access-9rs2f\" (UniqueName: \"kubernetes.io/projected/bdea8669-621e-4cae-8b36-d6057ba7955d-kube-api-access-9rs2f\") pod \"bdea8669-621e-4cae-8b36-d6057ba7955d\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " Oct 31 03:57:09.743414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.742803 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-xks-ca\") pod \"180074c2-4d96-4581-a301-a7e0e3b9ec27\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " Oct 31 03:57:09.743414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.742840 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-xks-ca\") pod \"bdea8669-621e-4cae-8b36-d6057ba7955d\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " Oct 31 03:57:09.743414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.742957 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-openshift-ca\") pod \"180074c2-4d96-4581-a301-a7e0e3b9ec27\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " Oct 31 03:57:09.743414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.743005 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-openshift-ca\") pod \"bdea8669-621e-4cae-8b36-d6057ba7955d\" (UID: \"bdea8669-621e-4cae-8b36-d6057ba7955d\") " Oct 31 03:57:09.743414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.743088 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180074c2-4d96-4581-a301-a7e0e3b9ec27-kube-api-access-ffb2l\" (UniqueName: \"kubernetes.io/projected/180074c2-4d96-4581-a301-a7e0e3b9ec27-kube-api-access-ffb2l\") pod \"180074c2-4d96-4581-a301-a7e0e3b9ec27\" (UID: \"180074c2-4d96-4581-a301-a7e0e3b9ec27\") " Oct 31 03:57:09.743667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.743502 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-openshift-ca" pod "180074c2-4d96-4581-a301-a7e0e3b9ec27" (UID: "180074c2-4d96-4581-a301-a7e0e3b9ec27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:09.743667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.743544 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-openshift-ca" pod "bdea8669-621e-4cae-8b36-d6057ba7955d" (UID: "bdea8669-621e-4cae-8b36-d6057ba7955d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:09.743667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.743635 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.745522 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.745495 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-xks-ca" pod "bdea8669-621e-4cae-8b36-d6057ba7955d" (UID: "bdea8669-621e-4cae-8b36-d6057ba7955d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:09.745725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.745700 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-xks-ca" pod "180074c2-4d96-4581-a301-a7e0e3b9ec27" (UID: "180074c2-4d96-4581-a301-a7e0e3b9ec27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:09.745725 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.745706 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdea8669-621e-4cae-8b36-d6057ba7955d-kube-api-access-9rs2f" pod "bdea8669-621e-4cae-8b36-d6057ba7955d" (UID: "bdea8669-621e-4cae-8b36-d6057ba7955d"). InnerVolumeSpecName "kube-api-access-9rs2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:09.745895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.745873 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180074c2-4d96-4581-a301-a7e0e3b9ec27-kube-api-access-ffb2l" pod "180074c2-4d96-4581-a301-a7e0e3b9ec27" (UID: "180074c2-4d96-4581-a301-a7e0e3b9ec27"). InnerVolumeSpecName "kube-api-access-ffb2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:09.844569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.844521 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.844569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.844565 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffb2l\" (UniqueName: \"kubernetes.io/projected/180074c2-4d96-4581-a301-a7e0e3b9ec27-kube-api-access-ffb2l\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.844569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.844575 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rs2f\" (UniqueName: \"kubernetes.io/projected/bdea8669-621e-4cae-8b36-d6057ba7955d-kube-api-access-9rs2f\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.844569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.844586 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/180074c2-4d96-4581-a301-a7e0e3b9ec27-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.844875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:09.844597 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdea8669-621e-4cae-8b36-d6057ba7955d-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:10.475456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:10.475416 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" Oct 31 03:57:10.475953 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:10.475417 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2" event={"ID":"180074c2-4d96-4581-a301-a7e0e3b9ec27","Type":"ContainerDied","Data":"73c6ede79ba5bebb0b43f0ff7563951c904078c14f20ea158413baf9c92abdc0"} Oct 31 03:57:10.475953 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:10.475560 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c6ede79ba5bebb0b43f0ff7563951c904078c14f20ea158413baf9c92abdc0" Oct 31 03:57:10.476972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:10.476949 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" Oct 31 03:57:10.477071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:10.476969 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf" event={"ID":"bdea8669-621e-4cae-8b36-d6057ba7955d","Type":"ContainerDied","Data":"2bcf7329e1c8daa3005576dc49b5e647dea366f7a3feb9bbe15bf0c918d857ab"} Oct 31 03:57:10.477071 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:10.477012 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bcf7329e1c8daa3005576dc49b5e647dea366f7a3feb9bbe15bf0c918d857ab" Oct 31 03:57:13.973877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:13.973835 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz"] Oct 31 03:57:13.980039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:13.980008 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk"] Oct 31 03:57:14.006234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.006105 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5"] Oct 31 03:57:14.181941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.181630 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz"] Oct 31 03:57:14.182414 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.182386 5520 cpu_manager.go:414] "containerMap: removing container" podUID="180074c2-4d96-4581-a301-a7e0e3b9ec27" containerName="revoke-keys" Oct 31 03:57:14.182548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.182416 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180074c2-4d96-4581-a301-a7e0e3b9ec27" containerName="revoke-keys" Oct 31 03:57:14.182548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.182432 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bdea8669-621e-4cae-8b36-d6057ba7955d" containerName="revoke-keys" Oct 31 03:57:14.182548 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.182442 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdea8669-621e-4cae-8b36-d6057ba7955d" containerName="revoke-keys" Oct 31 03:57:14.182721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.182611 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="180074c2-4d96-4581-a301-a7e0e3b9ec27" containerName="revoke-keys" Oct 31 03:57:14.182721 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.182623 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdea8669-621e-4cae-8b36-d6057ba7955d" containerName="revoke-keys" Oct 31 03:57:14.186033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.186007 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.193750 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.193722 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz"] Oct 31 03:57:14.197454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.197064 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-ab21e4\"/\"openshift-service-ca.crt\"" Oct 31 03:57:14.197454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.197225 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-ab21e4\"/\"default-dockercfg-2gkxr\"" Oct 31 03:57:14.197454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.197340 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-ab21e4\"/\"sub-test-model-ab21e4-kserve-self-signed-certs\"" Oct 31 03:57:14.197454 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.197422 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-ab21e4\"/\"kube-root-ca.crt\"" Oct 31 03:57:14.304533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.304434 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-dshm\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.304720 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.304648 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b538276b-f1ed-4ff6-a151-aadce56a20de-tls-certs\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.304807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.304766 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkpzc\" (UniqueName: \"kubernetes.io/projected/b538276b-f1ed-4ff6-a151-aadce56a20de-kube-api-access-pkpzc\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.304891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.304828 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-tmp-dir\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.304891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.304858 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-home\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.304891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.304882 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-model-cache\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.405927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.405882 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b538276b-f1ed-4ff6-a151-aadce56a20de-tls-certs\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.406831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.406804 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkpzc\" (UniqueName: \"kubernetes.io/projected/b538276b-f1ed-4ff6-a151-aadce56a20de-kube-api-access-pkpzc\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.407028 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.407014 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-tmp-dir\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.407172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.407157 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-home\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.407327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.407311 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-model-cache\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.407447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.407433 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-dshm\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.407997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.407975 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-tmp-dir\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.408579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.408547 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-model-cache\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.408984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.408965 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-home\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.428075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.427506 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-dshm\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.435262 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.435228 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b538276b-f1ed-4ff6-a151-aadce56a20de-tls-certs\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.440930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.440862 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkpzc\" (UniqueName: \"kubernetes.io/projected/b538276b-f1ed-4ff6-a151-aadce56a20de-kube-api-access-pkpzc\") pod \"sub-test-model-ab21e4-kserve-8487cc99-fw4dz\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.511569 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.511529 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:14.826040 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:14.826015 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz"] Oct 31 03:57:14.828869 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:57:14.828836 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb538276b_f1ed_4ff6_a151_aadce56a20de.slice/crio-71b605b782781ca6c2f4f8db4932f763b531e8de0c5ff98ed22cf0f3ea8075c8 WatchSource:0}: Error finding container 71b605b782781ca6c2f4f8db4932f763b531e8de0c5ff98ed22cf0f3ea8075c8: Status 404 returned error can't find the container with id 71b605b782781ca6c2f4f8db4932f763b531e8de0c5ff98ed22cf0f3ea8075c8 Oct 31 03:57:15.517266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:15.517227 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" event={"ID":"b538276b-f1ed-4ff6-a151-aadce56a20de","Type":"ContainerStarted","Data":"01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d"} Oct 31 03:57:15.517266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:15.517273 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" event={"ID":"b538276b-f1ed-4ff6-a151-aadce56a20de","Type":"ContainerStarted","Data":"71b605b782781ca6c2f4f8db4932f763b531e8de0c5ff98ed22cf0f3ea8075c8"} Oct 31 03:57:16.513431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:16.513393 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:23.818810 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.818702 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" podStartSLOduration=9.818687392 podStartE2EDuration="9.818687392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:15.533689747 +0000 UTC m=+2459.728212869" watchObservedRunningTime="2026-10-31 03:57:23.818687392 +0000 UTC m=+2468.013210490" Oct 31 03:57:23.821587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.821555 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f"] Oct 31 03:57:23.855419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.853524 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:23.885341 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.885214 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f"] Oct 31 03:57:23.926312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.926262 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th6j5\" (UniqueName: \"kubernetes.io/projected/7308a1b0-065f-46f8-b369-1f27972b5b2c-kube-api-access-th6j5\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:23.926529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.926413 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:23.926529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.926484 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:23.957806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.957762 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:57:23.957806 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:23.957801 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:57:24.027334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.027296 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:24.027549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.027407 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th6j5\" (UniqueName: \"kubernetes.io/projected/7308a1b0-065f-46f8-b369-1f27972b5b2c-kube-api-access-th6j5\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:24.027607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.027547 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:24.028395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.028361 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:24.030355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.030326 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:24.085281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.084552 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th6j5\" (UniqueName: \"kubernetes.io/projected/7308a1b0-065f-46f8-b369-1f27972b5b2c-kube-api-access-th6j5\") pod \"maas-api-revoke-sub-33abedb2-mkb8f\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:24.234962 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.234927 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:24.512331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.512300 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:24.524235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.524196 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f"] Oct 31 03:57:24.525498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.525475 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:24.526964 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:57:24.526941 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7308a1b0_065f_46f8_b369_1f27972b5b2c.slice/crio-d90cafbe69a5f8383882e7027969c5fe41f72b27a0175a49fd1a83cc5ec3f466 WatchSource:0}: Error finding container d90cafbe69a5f8383882e7027969c5fe41f72b27a0175a49fd1a83cc5ec3f466: Status 404 returned error can't find the container with id d90cafbe69a5f8383882e7027969c5fe41f72b27a0175a49fd1a83cc5ec3f466 Oct 31 03:57:24.587678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.587643 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" event={"ID":"7308a1b0-065f-46f8-b369-1f27972b5b2c","Type":"ContainerStarted","Data":"d90cafbe69a5f8383882e7027969c5fe41f72b27a0175a49fd1a83cc5ec3f466"} Oct 31 03:57:24.600752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:24.600728 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:57:25.595229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:25.595171 5520 generic.go:356] "Generic (PLEG): container finished" podID="7308a1b0-065f-46f8-b369-1f27972b5b2c" containerID="491ea8d1192fc00676d5e8fe15e7dfd10f55ae6dcc756cf1f981645280394e2f" exitCode=0 Oct 31 03:57:25.595734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:25.595275 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" event={"ID":"7308a1b0-065f-46f8-b369-1f27972b5b2c","Type":"ContainerDied","Data":"491ea8d1192fc00676d5e8fe15e7dfd10f55ae6dcc756cf1f981645280394e2f"} Oct 31 03:57:26.757160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.757133 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:26.865834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.865726 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-xks-ca\") pod \"7308a1b0-065f-46f8-b369-1f27972b5b2c\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " Oct 31 03:57:26.866020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.865875 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-openshift-ca\") pod \"7308a1b0-065f-46f8-b369-1f27972b5b2c\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " Oct 31 03:57:26.866020 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.865980 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7308a1b0-065f-46f8-b369-1f27972b5b2c-kube-api-access-th6j5\" (UniqueName: \"kubernetes.io/projected/7308a1b0-065f-46f8-b369-1f27972b5b2c-kube-api-access-th6j5\") pod \"7308a1b0-065f-46f8-b369-1f27972b5b2c\" (UID: \"7308a1b0-065f-46f8-b369-1f27972b5b2c\") " Oct 31 03:57:26.866395 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.866366 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-openshift-ca" pod "7308a1b0-065f-46f8-b369-1f27972b5b2c" (UID: "7308a1b0-065f-46f8-b369-1f27972b5b2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:26.866515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.866485 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:26.868297 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.868266 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-xks-ca" pod "7308a1b0-065f-46f8-b369-1f27972b5b2c" (UID: "7308a1b0-065f-46f8-b369-1f27972b5b2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:26.868438 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.868417 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7308a1b0-065f-46f8-b369-1f27972b5b2c-kube-api-access-th6j5" pod "7308a1b0-065f-46f8-b369-1f27972b5b2c" (UID: "7308a1b0-065f-46f8-b369-1f27972b5b2c"). InnerVolumeSpecName "kube-api-access-th6j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:26.967907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.967870 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7308a1b0-065f-46f8-b369-1f27972b5b2c-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:26.967907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:26.967904 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th6j5\" (UniqueName: \"kubernetes.io/projected/7308a1b0-065f-46f8-b369-1f27972b5b2c-kube-api-access-th6j5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:27.610934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:27.610899 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" event={"ID":"7308a1b0-065f-46f8-b369-1f27972b5b2c","Type":"ContainerDied","Data":"d90cafbe69a5f8383882e7027969c5fe41f72b27a0175a49fd1a83cc5ec3f466"} Oct 31 03:57:27.611121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:27.610955 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d90cafbe69a5f8383882e7027969c5fe41f72b27a0175a49fd1a83cc5ec3f466" Oct 31 03:57:27.611121 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:27.610911 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f" Oct 31 03:57:29.007266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.007176 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" containerID="cri-o://96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3" gracePeriod=45 Oct 31 03:57:29.008392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.008354 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" containerID="cri-o://dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216" gracePeriod=45 Oct 31 03:57:29.027498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.027460 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" containerID="cri-o://97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05" gracePeriod=45 Oct 31 03:57:29.316495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.313112 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:57:29.346298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.345469 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:57:29.348641 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.348612 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:57:29.398119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.398085 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-dshm\") pod \"e483869d-f714-4177-a566-4e6354c11fcf\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " Oct 31 03:57:29.398337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.398183 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-tmp-dir\") pod \"e483869d-f714-4177-a566-4e6354c11fcf\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " Oct 31 03:57:29.398337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.398241 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-model-cache\") pod \"e483869d-f714-4177-a566-4e6354c11fcf\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " Oct 31 03:57:29.398337 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.398309 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-home\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-home\") pod \"e483869d-f714-4177-a566-4e6354c11fcf\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " Oct 31 03:57:29.398531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.398350 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e483869d-f714-4177-a566-4e6354c11fcf-kube-api-access-wl72g\" (UniqueName: \"kubernetes.io/projected/e483869d-f714-4177-a566-4e6354c11fcf-kube-api-access-wl72g\") pod \"e483869d-f714-4177-a566-4e6354c11fcf\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " Oct 31 03:57:29.398531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.398433 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e483869d-f714-4177-a566-4e6354c11fcf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e483869d-f714-4177-a566-4e6354c11fcf-tls-certs\") pod \"e483869d-f714-4177-a566-4e6354c11fcf\" (UID: \"e483869d-f714-4177-a566-4e6354c11fcf\") " Oct 31 03:57:29.399965 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.399939 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-model-cache" pod "e483869d-f714-4177-a566-4e6354c11fcf" (UID: "e483869d-f714-4177-a566-4e6354c11fcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.401564 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.401540 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-tmp-dir" pod "e483869d-f714-4177-a566-4e6354c11fcf" (UID: "e483869d-f714-4177-a566-4e6354c11fcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.402106 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.401982 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-home" pod "e483869d-f714-4177-a566-4e6354c11fcf" (UID: "e483869d-f714-4177-a566-4e6354c11fcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.404908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.404877 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e483869d-f714-4177-a566-4e6354c11fcf-tls-certs" pod "e483869d-f714-4177-a566-4e6354c11fcf" (UID: "e483869d-f714-4177-a566-4e6354c11fcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:29.405634 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.405607 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e483869d-f714-4177-a566-4e6354c11fcf-kube-api-access-wl72g" pod "e483869d-f714-4177-a566-4e6354c11fcf" (UID: "e483869d-f714-4177-a566-4e6354c11fcf"). InnerVolumeSpecName "kube-api-access-wl72g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:29.408327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.408305 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-dshm" pod "e483869d-f714-4177-a566-4e6354c11fcf" (UID: "e483869d-f714-4177-a566-4e6354c11fcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.499536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499496 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-model-cache\") pod \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " Oct 31 03:57:29.499723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499548 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-dshm\") pod \"6b642464-78dd-415c-9a8a-35622d1abcbf\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " Oct 31 03:57:29.499723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499575 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-model-cache\") pod \"6b642464-78dd-415c-9a8a-35622d1abcbf\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " Oct 31 03:57:29.499723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499603 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-dshm\") pod \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " Oct 31 03:57:29.499723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499643 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b642464-78dd-415c-9a8a-35622d1abcbf-kube-api-access-x4mnz\" (UniqueName: \"kubernetes.io/projected/6b642464-78dd-415c-9a8a-35622d1abcbf-kube-api-access-x4mnz\") pod \"6b642464-78dd-415c-9a8a-35622d1abcbf\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " Oct 31 03:57:29.499723 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499687 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tmp-dir\") pod \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " Oct 31 03:57:29.500043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499768 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-model-cache" pod "af776a7f-f7b5-4392-a420-dcc54e7dd0df" (UID: "af776a7f-f7b5-4392-a420-dcc54e7dd0df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.500043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499800 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tls-certs\" (UniqueName: \"kubernetes.io/secret/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tls-certs\") pod \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " Oct 31 03:57:29.500043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499844 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af776a7f-f7b5-4392-a420-dcc54e7dd0df-kube-api-access-lmmvm\" (UniqueName: \"kubernetes.io/projected/af776a7f-f7b5-4392-a420-dcc54e7dd0df-kube-api-access-lmmvm\") pod \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " Oct 31 03:57:29.500043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499878 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-home\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-home\") pod \"6b642464-78dd-415c-9a8a-35622d1abcbf\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " Oct 31 03:57:29.500043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499936 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-tmp-dir\") pod \"6b642464-78dd-415c-9a8a-35622d1abcbf\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " Oct 31 03:57:29.500043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499991 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b642464-78dd-415c-9a8a-35622d1abcbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b642464-78dd-415c-9a8a-35622d1abcbf-tls-certs\") pod \"6b642464-78dd-415c-9a8a-35622d1abcbf\" (UID: \"6b642464-78dd-415c-9a8a-35622d1abcbf\") " Oct 31 03:57:29.500043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.500029 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-home\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-home\") pod \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\" (UID: \"af776a7f-f7b5-4392-a420-dcc54e7dd0df\") " Oct 31 03:57:29.500469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.499876 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-model-cache" pod "6b642464-78dd-415c-9a8a-35622d1abcbf" (UID: "6b642464-78dd-415c-9a8a-35622d1abcbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.500469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.500415 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tmp-dir" pod "af776a7f-f7b5-4392-a420-dcc54e7dd0df" (UID: "af776a7f-f7b5-4392-a420-dcc54e7dd0df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.500469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.500459 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-home" pod "6b642464-78dd-415c-9a8a-35622d1abcbf" (UID: "6b642464-78dd-415c-9a8a-35622d1abcbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.500658 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.500615 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-tmp-dir" pod "6b642464-78dd-415c-9a8a-35622d1abcbf" (UID: "6b642464-78dd-415c-9a8a-35622d1abcbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.500952 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-home" pod "af776a7f-f7b5-4392-a420-dcc54e7dd0df" (UID: "af776a7f-f7b5-4392-a420-dcc54e7dd0df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501282 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501304 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501319 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501333 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501348 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e483869d-f714-4177-a566-4e6354c11fcf-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501364 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl72g\" (UniqueName: \"kubernetes.io/projected/e483869d-f714-4177-a566-4e6354c11fcf-kube-api-access-wl72g\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501380 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501394 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e483869d-f714-4177-a566-4e6354c11fcf-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501408 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501425 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501440 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.501479 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.501454 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.502966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.502931 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b642464-78dd-415c-9a8a-35622d1abcbf-tls-certs" pod "6b642464-78dd-415c-9a8a-35622d1abcbf" (UID: "6b642464-78dd-415c-9a8a-35622d1abcbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:29.502966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.502940 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-dshm" pod "af776a7f-f7b5-4392-a420-dcc54e7dd0df" (UID: "af776a7f-f7b5-4392-a420-dcc54e7dd0df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.503219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.503179 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af776a7f-f7b5-4392-a420-dcc54e7dd0df-kube-api-access-lmmvm" pod "af776a7f-f7b5-4392-a420-dcc54e7dd0df" (UID: "af776a7f-f7b5-4392-a420-dcc54e7dd0df"). InnerVolumeSpecName "kube-api-access-lmmvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:29.503307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.503270 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tls-certs" pod "af776a7f-f7b5-4392-a420-dcc54e7dd0df" (UID: "af776a7f-f7b5-4392-a420-dcc54e7dd0df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:29.503510 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.503485 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b642464-78dd-415c-9a8a-35622d1abcbf-kube-api-access-x4mnz" pod "6b642464-78dd-415c-9a8a-35622d1abcbf" (UID: "6b642464-78dd-415c-9a8a-35622d1abcbf"). InnerVolumeSpecName "kube-api-access-x4mnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:29.504063 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.504016 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-dshm" pod "6b642464-78dd-415c-9a8a-35622d1abcbf" (UID: "6b642464-78dd-415c-9a8a-35622d1abcbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.602834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.602724 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b642464-78dd-415c-9a8a-35622d1abcbf-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.602834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.602767 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b642464-78dd-415c-9a8a-35622d1abcbf-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.602834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.602783 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af776a7f-f7b5-4392-a420-dcc54e7dd0df-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.602834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.602795 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4mnz\" (UniqueName: \"kubernetes.io/projected/6b642464-78dd-415c-9a8a-35622d1abcbf-kube-api-access-x4mnz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.602834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.602808 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af776a7f-f7b5-4392-a420-dcc54e7dd0df-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.602834 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.602821 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmmvm\" (UniqueName: \"kubernetes.io/projected/af776a7f-f7b5-4392-a420-dcc54e7dd0df-kube-api-access-lmmvm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.625549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.625514 5520 generic.go:356] "Generic (PLEG): container finished" podID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerID="97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05" exitCode=0 Oct 31 03:57:29.625745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.625558 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" Oct 31 03:57:29.625745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.625595 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" event={"ID":"6b642464-78dd-415c-9a8a-35622d1abcbf","Type":"ContainerDied","Data":"97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05"} Oct 31 03:57:29.625745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.625636 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5" event={"ID":"6b642464-78dd-415c-9a8a-35622d1abcbf","Type":"ContainerDied","Data":"d5ac0994210c82a15584765cdc69c8fdb5f6da585cb38bd0586a2bcdef71f65f"} Oct 31 03:57:29.625745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.625659 5520 scope.go:122] "RemoveContainer" containerID="97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05" Oct 31 03:57:29.626926 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.626903 5520 generic.go:356] "Generic (PLEG): container finished" podID="e483869d-f714-4177-a566-4e6354c11fcf" containerID="dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216" exitCode=0 Oct 31 03:57:29.627033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.626953 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" Oct 31 03:57:29.627033 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.626997 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" event={"ID":"e483869d-f714-4177-a566-4e6354c11fcf","Type":"ContainerDied","Data":"dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216"} Oct 31 03:57:29.627153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.627034 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk" event={"ID":"e483869d-f714-4177-a566-4e6354c11fcf","Type":"ContainerDied","Data":"56a61d4a992c2dd7e6bb7b75bbebf4aa3cf0b55a1abf51e6773d2c527a918b33"} Oct 31 03:57:29.628484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.628463 5520 generic.go:356] "Generic (PLEG): container finished" podID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerID="96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3" exitCode=0 Oct 31 03:57:29.628587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.628498 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" event={"ID":"af776a7f-f7b5-4392-a420-dcc54e7dd0df","Type":"ContainerDied","Data":"96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3"} Oct 31 03:57:29.628587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.628547 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" event={"ID":"af776a7f-f7b5-4392-a420-dcc54e7dd0df","Type":"ContainerDied","Data":"27a0eb4de4b01785b3c921c1d11654d447250254b546410b71fcb12b1e1f0fd4"} Oct 31 03:57:29.628587 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.628507 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz" Oct 31 03:57:29.636870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.636829 5520 scope.go:122] "RemoveContainer" containerID="97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05" Oct 31 03:57:29.637319 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:57:29.637263 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05\": container with ID starting with 97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05 not found: ID does not exist" containerID="97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05" Oct 31 03:57:29.637319 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.637298 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05"} err="failed to get container status \"97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05\": rpc error: code = NotFound desc = could not find container \"97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05\": container with ID starting with 97a052f3f3725bf84da033d70ee3f0ca7509db3c81ef4e680b9d518610255e05 not found: ID does not exist" Oct 31 03:57:29.637463 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.637325 5520 scope.go:122] "RemoveContainer" containerID="dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216" Oct 31 03:57:29.646046 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.646027 5520 scope.go:122] "RemoveContainer" containerID="dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216" Oct 31 03:57:29.646349 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:57:29.646330 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216\": container with ID starting with dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216 not found: ID does not exist" containerID="dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216" Oct 31 03:57:29.646417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.646358 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216"} err="failed to get container status \"dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216\": rpc error: code = NotFound desc = could not find container \"dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216\": container with ID starting with dfd62e351dafe8cb40806ae863b01b640724015d93abcc93c97330aef2911216 not found: ID does not exist" Oct 31 03:57:29.646417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.646374 5520 scope.go:122] "RemoveContainer" containerID="96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3" Oct 31 03:57:29.654417 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.654396 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5"] Oct 31 03:57:29.655037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.655025 5520 scope.go:122] "RemoveContainer" containerID="96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3" Oct 31 03:57:29.655317 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:57:29.655294 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3\": container with ID starting with 96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3 not found: ID does not exist" containerID="96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3" Oct 31 03:57:29.655410 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.655323 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3"} err="failed to get container status \"96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3\": rpc error: code = NotFound desc = could not find container \"96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3\": container with ID starting with 96ec55af2293a77d3ce4f33f20e0168fe5c3d81be4b6db56ec28d35a899b49f3 not found: ID does not exist" Oct 31 03:57:29.663018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.662987 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/simulator-w2-f20dc4-kserve-64fdbcf4bd-wfkz5"] Oct 31 03:57:29.674836 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.674811 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz"] Oct 31 03:57:29.678257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.678231 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-2-w2-f20dc4-kserve-858d745599-765tz"] Oct 31 03:57:29.689411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.689384 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk"] Oct 31 03:57:29.692952 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.692929 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/distinct-w2-f20dc4-kserve-6fbf86468-lhmjk"] Oct 31 03:57:29.917724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.917642 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" path="/var/lib/kubelet/pods/6b642464-78dd-415c-9a8a-35622d1abcbf/volumes" Oct 31 03:57:29.918075 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.918061 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" path="/var/lib/kubelet/pods/af776a7f-f7b5-4392-a420-dcc54e7dd0df/volumes" Oct 31 03:57:29.918481 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:29.918468 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e483869d-f714-4177-a566-4e6354c11fcf" path="/var/lib/kubelet/pods/e483869d-f714-4177-a566-4e6354c11fcf/volumes" Oct 31 03:57:31.350895 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:31.350858 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n"] Oct 31 03:57:31.351459 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:31.351097 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" containerID="cri-o://a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6" gracePeriod=30 Oct 31 03:57:36.621876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.621817 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:57:36.671597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.671564 5520 generic.go:356] "Generic (PLEG): container finished" podID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerID="a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6" exitCode=0 Oct 31 03:57:36.671769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.671628 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" Oct 31 03:57:36.671769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.671643 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" event={"ID":"1d842f7b-3ed4-402d-9c2a-2b7a2939c339","Type":"ContainerDied","Data":"a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6"} Oct 31 03:57:36.671769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.671707 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n" event={"ID":"1d842f7b-3ed4-402d-9c2a-2b7a2939c339","Type":"ContainerDied","Data":"8a4e1e24d4656cd7fe275050f556d3793aa8ad98c5e7ca23b2fccdbb214aed2b"} Oct 31 03:57:36.671769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.671731 5520 scope.go:122] "RemoveContainer" containerID="a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6" Oct 31 03:57:36.681016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.680992 5520 scope.go:122] "RemoveContainer" containerID="a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6" Oct 31 03:57:36.681387 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:57:36.681364 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6\": container with ID starting with a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6 not found: ID does not exist" containerID="a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6" Oct 31 03:57:36.681457 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.681395 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6"} err="failed to get container status \"a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6\": rpc error: code = NotFound desc = could not find container \"a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6\": container with ID starting with a46ac894157cdaabe502f185524d296245b1faecb691c505ff62dcb78f497ca6 not found: ID does not exist" Oct 31 03:57:36.710580 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710549 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-socket\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.710744 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710634 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-kube-api-access-dxzkg\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-kube-api-access-dxzkg\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.710744 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710678 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-certs\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.710744 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710711 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-podinfo\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.710928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710769 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-data\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.710928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710833 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-envoy\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.710928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710868 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-token\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-token\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.710928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710863 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-socket" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:36.710928 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710917 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istiod-ca-cert\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.711179 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.710944 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-credential-socket\") pod \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\" (UID: \"1d842f7b-3ed4-402d-9c2a-2b7a2939c339\") " Oct 31 03:57:36.711656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.711326 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-certs" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:36.711656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.711392 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.711656 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.711540 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-credential-socket" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:36.711870 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.711753 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istiod-ca-cert" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:36.712130 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.712096 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-data" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:36.714023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.713580 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-podinfo" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:36.714023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.713663 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-kube-api-access-dxzkg" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "kube-api-access-dxzkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:36.714778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.714100 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-envoy" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:36.715502 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.714956 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-token" pod "1d842f7b-3ed4-402d-9c2a-2b7a2939c339" (UID: "1d842f7b-3ed4-402d-9c2a-2b7a2939c339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:36.812313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812275 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxzkg\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-kube-api-access-dxzkg\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.812313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812305 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.812313 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812315 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.812549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812323 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.812549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812332 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.812549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812341 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.812549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812349 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:36.812549 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:36.812357 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d842f7b-3ed4-402d-9c2a-2b7a2939c339-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:37.011770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:37.011733 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n"] Oct 31 03:57:37.035307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:37.035279 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-zdx4n"] Oct 31 03:57:37.915826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:37.915792 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" path="/var/lib/kubelet/pods/1d842f7b-3ed4-402d-9c2a-2b7a2939c339/volumes" Oct 31 03:57:39.358465 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:57:39.358429 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:57:41.623821 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:41.623774 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h"] Oct 31 03:57:41.624383 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:41.624075 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" podUID="92500990-fcda-4877-8297-c8d53af11174" containerName="maas-api" containerID="cri-o://2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea" gracePeriod=30 Oct 31 03:57:41.977109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:41.977080 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:57:42.074883 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.074844 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92500990-fcda-4877-8297-c8d53af11174-kube-api-access-xqtnd\" (UniqueName: \"kubernetes.io/projected/92500990-fcda-4877-8297-c8d53af11174-kube-api-access-xqtnd\") pod \"92500990-fcda-4877-8297-c8d53af11174\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " Oct 31 03:57:42.075053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.074900 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-maas-api-tls\") pod \"92500990-fcda-4877-8297-c8d53af11174\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " Oct 31 03:57:42.075053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.074933 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-metrics-certs\") pod \"92500990-fcda-4877-8297-c8d53af11174\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " Oct 31 03:57:42.075224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.075113 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92500990-fcda-4877-8297-c8d53af11174-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92500990-fcda-4877-8297-c8d53af11174-openshift-service-ca\") pod \"92500990-fcda-4877-8297-c8d53af11174\" (UID: \"92500990-fcda-4877-8297-c8d53af11174\") " Oct 31 03:57:42.075722 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.075691 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92500990-fcda-4877-8297-c8d53af11174-openshift-service-ca" pod "92500990-fcda-4877-8297-c8d53af11174" (UID: "92500990-fcda-4877-8297-c8d53af11174"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:42.077450 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.077423 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-maas-api-tls" pod "92500990-fcda-4877-8297-c8d53af11174" (UID: "92500990-fcda-4877-8297-c8d53af11174"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:42.077581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.077564 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92500990-fcda-4877-8297-c8d53af11174-kube-api-access-xqtnd" pod "92500990-fcda-4877-8297-c8d53af11174" (UID: "92500990-fcda-4877-8297-c8d53af11174"). InnerVolumeSpecName "kube-api-access-xqtnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:42.077727 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.077709 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-metrics-certs" pod "92500990-fcda-4877-8297-c8d53af11174" (UID: "92500990-fcda-4877-8297-c8d53af11174"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:42.176633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.176592 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqtnd\" (UniqueName: \"kubernetes.io/projected/92500990-fcda-4877-8297-c8d53af11174-kube-api-access-xqtnd\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:42.176633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.176628 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-maas-api-tls\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:42.176633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.176638 5520 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92500990-fcda-4877-8297-c8d53af11174-metrics-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:42.176885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.176649 5520 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92500990-fcda-4877-8297-c8d53af11174-openshift-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:42.709979 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.709942 5520 generic.go:356] "Generic (PLEG): container finished" podID="92500990-fcda-4877-8297-c8d53af11174" containerID="2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea" exitCode=0 Oct 31 03:57:42.710473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.710002 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" Oct 31 03:57:42.710473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.710038 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" event={"ID":"92500990-fcda-4877-8297-c8d53af11174","Type":"ContainerDied","Data":"2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea"} Oct 31 03:57:42.710473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.710096 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h" event={"ID":"92500990-fcda-4877-8297-c8d53af11174","Type":"ContainerDied","Data":"c7c280da0d6489d169cd44a2e137b70271d59aac6f8cc3efbff6bb4151de29d5"} Oct 31 03:57:42.710473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.710120 5520 scope.go:122] "RemoveContainer" containerID="2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea" Oct 31 03:57:42.725291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.725269 5520 scope.go:122] "RemoveContainer" containerID="2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea" Oct 31 03:57:42.725579 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:57:42.725556 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea\": container with ID starting with 2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea not found: ID does not exist" containerID="2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea" Oct 31 03:57:42.725633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.725589 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea"} err="failed to get container status \"2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea\": rpc error: code = NotFound desc = could not find container \"2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea\": container with ID starting with 2cfc312ee1b1aa5f45b0605c46a0b70eca97bb74b98bca9b59bdd6142f3eccea not found: ID does not exist" Oct 31 03:57:42.735175 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.735154 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h"] Oct 31 03:57:42.739773 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:42.739753 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f20dc4-6cb448d544-9jw2h"] Oct 31 03:57:43.917083 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:43.917044 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92500990-fcda-4877-8297-c8d53af11174" path="/var/lib/kubelet/pods/92500990-fcda-4877-8297-c8d53af11174/volumes" Oct 31 03:57:49.618576 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:49.618538 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp"] Oct 31 03:57:49.619323 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:49.619291 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" containerID="cri-o://7796aed021dfda83fa6f61fe8b0e0d1f4ebe218ef2a31cac33b4bb1bc1c44e7a" gracePeriod=30 Oct 31 03:57:53.480984 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:53.480937 5520 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:57:53.481471 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:53.481010 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:57:54.787692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:54.787642 5520 generic.go:356] "Generic (PLEG): container finished" podID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerID="7796aed021dfda83fa6f61fe8b0e0d1f4ebe218ef2a31cac33b4bb1bc1c44e7a" exitCode=0 Oct 31 03:57:54.788180 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:54.787716 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" event={"ID":"ee92ee0d-060d-4e10-a543-7302c0e7e5e1","Type":"ContainerDied","Data":"7796aed021dfda83fa6f61fe8b0e0d1f4ebe218ef2a31cac33b4bb1bc1c44e7a"} Oct 31 03:57:54.890160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:54.890080 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:57:55.023700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.023669 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-podinfo\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.023906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.023714 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-kube-api-access-st22f\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-kube-api-access-st22f\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.023906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.023747 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-socket\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.023906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.023780 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-certs\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.023906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.023883 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-envoy\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.024147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.023948 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-credential-socket\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.024147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024005 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-data\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.024147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024038 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istiod-ca-cert\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.024147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024097 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-token\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-token\") pod \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\" (UID: \"ee92ee0d-060d-4e10-a543-7302c0e7e5e1\") " Oct 31 03:57:55.024147 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024124 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-socket" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:55.024466 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024150 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-certs" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:55.024570 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024544 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-credential-socket" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:55.024633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024566 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istiod-ca-cert" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:55.024705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.024333 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-data" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:55.025667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.025647 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.025667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.025669 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.025856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.025680 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.025856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.025690 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.025856 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.025698 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.026758 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.026724 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-token" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:55.026874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.026764 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-kube-api-access-st22f" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "kube-api-access-st22f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:55.027446 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.027424 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-envoy" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:55.027741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.027712 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-podinfo" pod "ee92ee0d-060d-4e10-a543-7302c0e7e5e1" (UID: "ee92ee0d-060d-4e10-a543-7302c0e7e5e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:55.126787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.126745 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.126787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.126777 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.127014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.126793 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st22f\" (UniqueName: \"kubernetes.io/projected/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-kube-api-access-st22f\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.127014 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.126808 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee92ee0d-060d-4e10-a543-7302c0e7e5e1-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:55.798067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.798031 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" Oct 31 03:57:55.798839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.798074 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp" event={"ID":"ee92ee0d-060d-4e10-a543-7302c0e7e5e1","Type":"ContainerDied","Data":"b49c3eaa1a430fb9e73e458bb8464bb13ab62cbce29f08fad91b026fafaa6f59"} Oct 31 03:57:55.798839 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.798150 5520 scope.go:122] "RemoveContainer" containerID="7796aed021dfda83fa6f61fe8b0e0d1f4ebe218ef2a31cac33b4bb1bc1c44e7a" Oct 31 03:57:55.829001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.828971 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp"] Oct 31 03:57:55.833566 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.833542 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-m85gp"] Oct 31 03:57:55.917136 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:57:55.917088 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" path="/var/lib/kubelet/pods/ee92ee0d-060d-4e10-a543-7302c0e7e5e1/volumes" Oct 31 03:58:09.358352 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:58:09.358315 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:58:27.316672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.316588 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7279-859d10"] Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317046 5520 cpu_manager.go:414] "containerMap: removing container" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317060 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317079 5520 cpu_manager.go:414] "containerMap: removing container" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317084 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317092 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7308a1b0-065f-46f8-b369-1f27972b5b2c" containerName="revoke-keys" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317098 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7308a1b0-065f-46f8-b369-1f27972b5b2c" containerName="revoke-keys" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317105 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317110 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317117 5520 cpu_manager.go:414] "containerMap: removing container" podUID="92500990-fcda-4877-8297-c8d53af11174" containerName="maas-api" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317124 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92500990-fcda-4877-8297-c8d53af11174" containerName="maas-api" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317129 5520 cpu_manager.go:414] "containerMap: removing container" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317133 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317141 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" Oct 31 03:58:27.317172 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317148 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" Oct 31 03:58:27.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317249 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="af776a7f-f7b5-4392-a420-dcc54e7dd0df" containerName="main" Oct 31 03:58:27.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317259 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d842f7b-3ed4-402d-9c2a-2b7a2939c339" containerName="istio-proxy" Oct 31 03:58:27.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317265 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7308a1b0-065f-46f8-b369-1f27972b5b2c" containerName="revoke-keys" Oct 31 03:58:27.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317272 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee92ee0d-060d-4e10-a543-7302c0e7e5e1" containerName="istio-proxy" Oct 31 03:58:27.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317278 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="92500990-fcda-4877-8297-c8d53af11174" containerName="maas-api" Oct 31 03:58:27.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317283 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b642464-78dd-415c-9a8a-35622d1abcbf" containerName="main" Oct 31 03:58:27.317651 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.317289 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e483869d-f714-4177-a566-4e6354c11fcf" containerName="main" Oct 31 03:58:27.321908 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.321890 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7279-859d10" Oct 31 03:58:27.325359 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.325333 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xd6kv\"" Oct 31 03:58:27.326625 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.326599 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7279-859d10"] Oct 31 03:58:27.403011 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.402977 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk9nf\" (UniqueName: \"kubernetes.io/projected/01bdf991-fc64-438f-bba9-b76552613683-kube-api-access-mk9nf\") pod \"test-curl-7279-859d10\" (UID: \"01bdf991-fc64-438f-bba9-b76552613683\") " pod="opendatahub/test-curl-7279-859d10" Oct 31 03:58:27.506070 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.506024 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk9nf\" (UniqueName: \"kubernetes.io/projected/01bdf991-fc64-438f-bba9-b76552613683-kube-api-access-mk9nf\") pod \"test-curl-7279-859d10\" (UID: \"01bdf991-fc64-438f-bba9-b76552613683\") " pod="opendatahub/test-curl-7279-859d10" Oct 31 03:58:27.518547 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.518511 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk9nf\" (UniqueName: \"kubernetes.io/projected/01bdf991-fc64-438f-bba9-b76552613683-kube-api-access-mk9nf\") pod \"test-curl-7279-859d10\" (UID: \"01bdf991-fc64-438f-bba9-b76552613683\") " pod="opendatahub/test-curl-7279-859d10" Oct 31 03:58:27.636072 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.635985 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7279-859d10" Oct 31 03:58:27.916048 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:58:27.916014 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01bdf991_fc64_438f_bba9_b76552613683.slice/crio-6468295038b9ba265943b8546f515952d786a43312ff16448d4923eef50bc67d WatchSource:0}: Error finding container 6468295038b9ba265943b8546f515952d786a43312ff16448d4923eef50bc67d: Status 404 returned error can't find the container with id 6468295038b9ba265943b8546f515952d786a43312ff16448d4923eef50bc67d Oct 31 03:58:27.919195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.919174 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7279-859d10"] Oct 31 03:58:27.971648 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:27.971619 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7279-859d10" event={"ID":"01bdf991-fc64-438f-bba9-b76552613683","Type":"ContainerStarted","Data":"6468295038b9ba265943b8546f515952d786a43312ff16448d4923eef50bc67d"} Oct 31 03:58:28.979682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:28.979644 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7279-859d10" event={"ID":"01bdf991-fc64-438f-bba9-b76552613683","Type":"ContainerStarted","Data":"b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635"} Oct 31 03:58:28.994804 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:28.994746 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7279-859d10" podStartSLOduration=1.994733451 podStartE2EDuration="1.994733451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:28.992766157 +0000 UTC m=+2533.187289288" watchObservedRunningTime="2026-10-31 03:58:28.994733451 +0000 UTC m=+2533.189256571" Oct 31 03:58:29.663857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.663820 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7279-859d10"] Oct 31 03:58:29.667732 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.667703 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7279-859d10"] Oct 31 03:58:29.871526 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.871482 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q"] Oct 31 03:58:29.871931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.871913 5520 cpu_manager.go:414] "containerMap: removing container" podUID="01bdf991-fc64-438f-bba9-b76552613683" containerName="test-curl-7279-859d10" Oct 31 03:58:29.872027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.871932 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01bdf991-fc64-438f-bba9-b76552613683" containerName="test-curl-7279-859d10" Oct 31 03:58:29.872087 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.872040 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="01bdf991-fc64-438f-bba9-b76552613683" containerName="test-curl-7279-859d10" Oct 31 03:58:29.875399 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.875376 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:29.888970 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.888947 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:58:29.889597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.889575 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:58:29.889751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.889575 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:58:29.889751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.889611 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:58:29.893405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:29.893376 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q"] Oct 31 03:58:30.042797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.042749 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlpv\" (UniqueName: \"kubernetes.io/projected/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-kube-api-access-mhlpv\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.043325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.042900 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.043325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.042982 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.144277 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.144246 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlpv\" (UniqueName: \"kubernetes.io/projected/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-kube-api-access-mhlpv\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.144464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.144316 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.144464 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.144353 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.144951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.144932 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.146917 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.146897 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.155753 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.155726 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlpv\" (UniqueName: \"kubernetes.io/projected/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-kube-api-access-mhlpv\") pod \"maas-api-revoke-sub-ddfc873c-r9h5q\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.188800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.188768 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:30.473749 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.473718 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q"] Oct 31 03:58:30.476786 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:58:30.476751 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefab1f60_b8e4_470a_b3c9_7212d5eb1ba1.slice/crio-6e72a6d2751693607258bcd032d6c74fc784a8b1256ead64cee5997fe39d875e WatchSource:0}: Error finding container 6e72a6d2751693607258bcd032d6c74fc784a8b1256ead64cee5997fe39d875e: Status 404 returned error can't find the container with id 6e72a6d2751693607258bcd032d6c74fc784a8b1256ead64cee5997fe39d875e Oct 31 03:58:31.003146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.996487 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7279-859d10" podUID="01bdf991-fc64-438f-bba9-b76552613683" containerName="test-curl-7279-859d10" containerID="cri-o://b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635" gracePeriod=2 Oct 31 03:58:31.003146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.997508 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" event={"ID":"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1","Type":"ContainerStarted","Data":"fda761e8e35c0cfddea5bc25ff73ab8d24085b4e88abb3e78d3e50be0de5e159"} Oct 31 03:58:31.003146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:30.997564 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" event={"ID":"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1","Type":"ContainerStarted","Data":"6e72a6d2751693607258bcd032d6c74fc784a8b1256ead64cee5997fe39d875e"} Oct 31 03:58:31.020212 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:31.020127 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" podStartSLOduration=2.020105482 podStartE2EDuration="2.020105482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:31.016505061 +0000 UTC m=+2535.211028205" watchObservedRunningTime="2026-10-31 03:58:31.020105482 +0000 UTC m=+2535.214628604" Oct 31 03:58:32.003631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:32.003591 5520 generic.go:356] "Generic (PLEG): container finished" podID="efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" containerID="fda761e8e35c0cfddea5bc25ff73ab8d24085b4e88abb3e78d3e50be0de5e159" exitCode=0 Oct 31 03:58:32.004178 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:32.003635 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" event={"ID":"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1","Type":"ContainerDied","Data":"fda761e8e35c0cfddea5bc25ff73ab8d24085b4e88abb3e78d3e50be0de5e159"} Oct 31 03:58:33.184774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.184744 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:33.185392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.185102 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-kube-api-access-mhlpv\" (UniqueName: \"kubernetes.io/projected/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-kube-api-access-mhlpv\") pod \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " Oct 31 03:58:33.185392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.185146 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-xks-ca\") pod \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " Oct 31 03:58:33.185392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.185297 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-openshift-ca\") pod \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\" (UID: \"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1\") " Oct 31 03:58:33.185802 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.185771 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-openshift-ca" pod "efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" (UID: "efab1f60-b8e4-470a-b3c9-7212d5eb1ba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:33.188042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.188016 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-xks-ca" pod "efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" (UID: "efab1f60-b8e4-470a-b3c9-7212d5eb1ba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:33.188129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.188058 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-kube-api-access-mhlpv" pod "efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" (UID: "efab1f60-b8e4-470a-b3c9-7212d5eb1ba1"). InnerVolumeSpecName "kube-api-access-mhlpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:33.226543 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.226513 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7279-859d10" Oct 31 03:58:33.286684 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.286601 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01bdf991-fc64-438f-bba9-b76552613683-kube-api-access-mk9nf\" (UniqueName: \"kubernetes.io/projected/01bdf991-fc64-438f-bba9-b76552613683-kube-api-access-mk9nf\") pod \"01bdf991-fc64-438f-bba9-b76552613683\" (UID: \"01bdf991-fc64-438f-bba9-b76552613683\") " Oct 31 03:58:33.286854 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.286838 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.286899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.286853 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlpv\" (UniqueName: \"kubernetes.io/projected/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-kube-api-access-mhlpv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.286899 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.286863 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.288858 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.288831 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01bdf991-fc64-438f-bba9-b76552613683-kube-api-access-mk9nf" pod "01bdf991-fc64-438f-bba9-b76552613683" (UID: "01bdf991-fc64-438f-bba9-b76552613683"). InnerVolumeSpecName "kube-api-access-mk9nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:33.387692 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.387649 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk9nf\" (UniqueName: \"kubernetes.io/projected/01bdf991-fc64-438f-bba9-b76552613683-kube-api-access-mk9nf\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:33.916666 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:33.916632 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01bdf991-fc64-438f-bba9-b76552613683" path="/var/lib/kubelet/pods/01bdf991-fc64-438f-bba9-b76552613683/volumes" Oct 31 03:58:34.031334 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.031303 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" Oct 31 03:58:34.031532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.031331 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q" event={"ID":"efab1f60-b8e4-470a-b3c9-7212d5eb1ba1","Type":"ContainerDied","Data":"6e72a6d2751693607258bcd032d6c74fc784a8b1256ead64cee5997fe39d875e"} Oct 31 03:58:34.031532 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.031389 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e72a6d2751693607258bcd032d6c74fc784a8b1256ead64cee5997fe39d875e" Oct 31 03:58:34.032919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.032891 5520 generic.go:356] "Generic (PLEG): container finished" podID="01bdf991-fc64-438f-bba9-b76552613683" containerID="b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635" exitCode=137 Oct 31 03:58:34.033053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.032978 5520 scope.go:122] "RemoveContainer" containerID="b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635" Oct 31 03:58:34.033053 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.032992 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7279-859d10" Oct 31 03:58:34.043797 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.043779 5520 scope.go:122] "RemoveContainer" containerID="b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635" Oct 31 03:58:34.044134 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:58:34.044097 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635\": container with ID starting with b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635 not found: ID does not exist" containerID="b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635" Oct 31 03:58:34.044263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.044137 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635"} err="failed to get container status \"b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635\": rpc error: code = NotFound desc = could not find container \"b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635\": container with ID starting with b63d2100390bc775c43f2d49e35f6449e4aa1af10104f5b8f752349c0a30c635 not found: ID does not exist" Oct 31 03:58:34.727158 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.727120 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj"] Oct 31 03:58:34.727657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.727574 5520 cpu_manager.go:414] "containerMap: removing container" podUID="efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" containerName="revoke-keys" Oct 31 03:58:34.727657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.727588 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" containerName="revoke-keys" Oct 31 03:58:34.727791 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.727663 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" containerName="revoke-keys" Oct 31 03:58:34.732093 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.732075 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.739365 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.739271 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj"] Oct 31 03:58:34.740242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.740192 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:58:34.741235 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.741184 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:58:34.741352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.741272 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:58:34.741401 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.741350 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:58:34.802670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.802629 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.802881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.802789 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.802881 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.802835 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdh86\" (UniqueName: \"kubernetes.io/projected/6d899d87-d746-4878-b5e0-4b7c30199599-kube-api-access-hdh86\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.904294 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.904246 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.904577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.904314 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdh86\" (UniqueName: \"kubernetes.io/projected/6d899d87-d746-4878-b5e0-4b7c30199599-kube-api-access-hdh86\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.904577 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.904531 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.905009 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.904984 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.907253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.907226 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:34.912016 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:34.911992 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdh86\" (UniqueName: \"kubernetes.io/projected/6d899d87-d746-4878-b5e0-4b7c30199599-kube-api-access-hdh86\") pod \"maas-api-revoke-sub-f309a45b-jbktj\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:35.047715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:35.047635 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:35.350581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:35.350508 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj"] Oct 31 03:58:35.352937 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:58:35.352908 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d899d87_d746_4878_b5e0_4b7c30199599.slice/crio-ebd380da544de612956aaded76017a1597b57f3f6f971c8ac79d3f111e6b1fee WatchSource:0}: Error finding container ebd380da544de612956aaded76017a1597b57f3f6f971c8ac79d3f111e6b1fee: Status 404 returned error can't find the container with id ebd380da544de612956aaded76017a1597b57f3f6f971c8ac79d3f111e6b1fee Oct 31 03:58:36.046835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:36.046791 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" event={"ID":"6d899d87-d746-4878-b5e0-4b7c30199599","Type":"ContainerStarted","Data":"cb60710623cfda4ebfe4f19a61da597bf45e2c556922db3e727e8a28c07c4b19"} Oct 31 03:58:36.047320 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:36.046852 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" event={"ID":"6d899d87-d746-4878-b5e0-4b7c30199599","Type":"ContainerStarted","Data":"ebd380da544de612956aaded76017a1597b57f3f6f971c8ac79d3f111e6b1fee"} Oct 31 03:58:36.062770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:36.062719 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" podStartSLOduration=2.062707311 podStartE2EDuration="2.062707311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:36.062080508 +0000 UTC m=+2540.256603629" watchObservedRunningTime="2026-10-31 03:58:36.062707311 +0000 UTC m=+2540.257230435" Oct 31 03:58:37.054946 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:37.054913 5520 generic.go:356] "Generic (PLEG): container finished" podID="6d899d87-d746-4878-b5e0-4b7c30199599" containerID="cb60710623cfda4ebfe4f19a61da597bf45e2c556922db3e727e8a28c07c4b19" exitCode=0 Oct 31 03:58:37.055498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:37.054996 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" event={"ID":"6d899d87-d746-4878-b5e0-4b7c30199599","Type":"ContainerDied","Data":"cb60710623cfda4ebfe4f19a61da597bf45e2c556922db3e727e8a28c07c4b19"} Oct 31 03:58:38.191035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.191003 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:38.256961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.256931 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d899d87-d746-4878-b5e0-4b7c30199599-kube-api-access-hdh86\" (UniqueName: \"kubernetes.io/projected/6d899d87-d746-4878-b5e0-4b7c30199599-kube-api-access-hdh86\") pod \"6d899d87-d746-4878-b5e0-4b7c30199599\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " Oct 31 03:58:38.257160 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.257000 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-xks-ca\") pod \"6d899d87-d746-4878-b5e0-4b7c30199599\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " Oct 31 03:58:38.257266 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.257180 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-openshift-ca\") pod \"6d899d87-d746-4878-b5e0-4b7c30199599\" (UID: \"6d899d87-d746-4878-b5e0-4b7c30199599\") " Oct 31 03:58:38.257518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.257498 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-openshift-ca" pod "6d899d87-d746-4878-b5e0-4b7c30199599" (UID: "6d899d87-d746-4878-b5e0-4b7c30199599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:38.257631 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.257605 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:38.259283 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.259252 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-xks-ca" pod "6d899d87-d746-4878-b5e0-4b7c30199599" (UID: "6d899d87-d746-4878-b5e0-4b7c30199599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:38.259397 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.259326 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d899d87-d746-4878-b5e0-4b7c30199599-kube-api-access-hdh86" pod "6d899d87-d746-4878-b5e0-4b7c30199599" (UID: "6d899d87-d746-4878-b5e0-4b7c30199599"). InnerVolumeSpecName "kube-api-access-hdh86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:38.359371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.359291 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdh86\" (UniqueName: \"kubernetes.io/projected/6d899d87-d746-4878-b5e0-4b7c30199599-kube-api-access-hdh86\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:38.359371 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:38.359329 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d899d87-d746-4878-b5e0-4b7c30199599-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:39.067146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:39.067116 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" Oct 31 03:58:39.067348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:39.067140 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj" event={"ID":"6d899d87-d746-4878-b5e0-4b7c30199599","Type":"ContainerDied","Data":"ebd380da544de612956aaded76017a1597b57f3f6f971c8ac79d3f111e6b1fee"} Oct 31 03:58:39.067348 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:39.067196 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd380da544de612956aaded76017a1597b57f3f6f971c8ac79d3f111e6b1fee" Oct 31 03:58:39.363950 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:58:39.363853 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:58:40.078132 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.078099 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6"] Oct 31 03:58:40.078724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.078704 5520 cpu_manager.go:414] "containerMap: removing container" podUID="6d899d87-d746-4878-b5e0-4b7c30199599" containerName="revoke-keys" Oct 31 03:58:40.078787 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.078730 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d899d87-d746-4878-b5e0-4b7c30199599" containerName="revoke-keys" Oct 31 03:58:40.078878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.078867 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d899d87-d746-4878-b5e0-4b7c30199599" containerName="revoke-keys" Oct 31 03:58:40.082473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.082454 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.087153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.087127 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:58:40.087287 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.087241 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:58:40.089966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.089932 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6"] Oct 31 03:58:40.151486 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.151459 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:58:40.151681 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.151473 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:58:40.179109 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.179070 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfjf\" (UniqueName: \"kubernetes.io/projected/7ac60650-0f41-4ee2-9f17-22435a8897b2-kube-api-access-dcfjf\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.179286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.179122 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.179286 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.179244 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.280475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.280436 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfjf\" (UniqueName: \"kubernetes.io/projected/7ac60650-0f41-4ee2-9f17-22435a8897b2-kube-api-access-dcfjf\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.280670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.280490 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.280670 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.280526 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.281411 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.281386 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.282944 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.282924 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.287887 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.287863 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfjf\" (UniqueName: \"kubernetes.io/projected/7ac60650-0f41-4ee2-9f17-22435a8897b2-kube-api-access-dcfjf\") pod \"maas-api-revoke-sub-d8df7f75-z5dm6\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.397607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.397521 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:40.671585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:40.671558 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6"] Oct 31 03:58:40.674143 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:58:40.674109 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac60650_0f41_4ee2_9f17_22435a8897b2.slice/crio-6886382910ecc79007db0a6af7f4750def99d623a3105e52f0af0d90a344a800 WatchSource:0}: Error finding container 6886382910ecc79007db0a6af7f4750def99d623a3105e52f0af0d90a344a800: Status 404 returned error can't find the container with id 6886382910ecc79007db0a6af7f4750def99d623a3105e52f0af0d90a344a800 Oct 31 03:58:41.086472 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:41.086428 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" event={"ID":"7ac60650-0f41-4ee2-9f17-22435a8897b2","Type":"ContainerStarted","Data":"16390e848426f107409014aca9d1a8b187732eb822e33ddc19e02b689be30f7b"} Oct 31 03:58:41.087312 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:41.087271 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" event={"ID":"7ac60650-0f41-4ee2-9f17-22435a8897b2","Type":"ContainerStarted","Data":"6886382910ecc79007db0a6af7f4750def99d623a3105e52f0af0d90a344a800"} Oct 31 03:58:41.104655 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:41.104600 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" podStartSLOduration=1.10458683 podStartE2EDuration="1.10458683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:41.102521004 +0000 UTC m=+2545.297044136" watchObservedRunningTime="2026-10-31 03:58:41.10458683 +0000 UTC m=+2545.299109991" Oct 31 03:58:42.095222 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:42.095174 5520 generic.go:356] "Generic (PLEG): container finished" podID="7ac60650-0f41-4ee2-9f17-22435a8897b2" containerID="16390e848426f107409014aca9d1a8b187732eb822e33ddc19e02b689be30f7b" exitCode=0 Oct 31 03:58:42.095654 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:42.095267 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" event={"ID":"7ac60650-0f41-4ee2-9f17-22435a8897b2","Type":"ContainerDied","Data":"16390e848426f107409014aca9d1a8b187732eb822e33ddc19e02b689be30f7b"} Oct 31 03:58:43.241476 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.241450 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:43.311232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.311168 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-openshift-ca\") pod \"7ac60650-0f41-4ee2-9f17-22435a8897b2\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " Oct 31 03:58:43.311416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.311316 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac60650-0f41-4ee2-9f17-22435a8897b2-kube-api-access-dcfjf\" (UniqueName: \"kubernetes.io/projected/7ac60650-0f41-4ee2-9f17-22435a8897b2-kube-api-access-dcfjf\") pod \"7ac60650-0f41-4ee2-9f17-22435a8897b2\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " Oct 31 03:58:43.311416 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.311384 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-xks-ca\") pod \"7ac60650-0f41-4ee2-9f17-22435a8897b2\" (UID: \"7ac60650-0f41-4ee2-9f17-22435a8897b2\") " Oct 31 03:58:43.311626 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.311597 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-openshift-ca" pod "7ac60650-0f41-4ee2-9f17-22435a8897b2" (UID: "7ac60650-0f41-4ee2-9f17-22435a8897b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:43.311786 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.311765 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:43.313724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.313697 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-xks-ca" pod "7ac60650-0f41-4ee2-9f17-22435a8897b2" (UID: "7ac60650-0f41-4ee2-9f17-22435a8897b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:43.313827 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.313744 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac60650-0f41-4ee2-9f17-22435a8897b2-kube-api-access-dcfjf" pod "7ac60650-0f41-4ee2-9f17-22435a8897b2" (UID: "7ac60650-0f41-4ee2-9f17-22435a8897b2"). InnerVolumeSpecName "kube-api-access-dcfjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:43.413153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.413062 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfjf\" (UniqueName: \"kubernetes.io/projected/7ac60650-0f41-4ee2-9f17-22435a8897b2-kube-api-access-dcfjf\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:43.413153 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:43.413106 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac60650-0f41-4ee2-9f17-22435a8897b2-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:44.109955 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.109917 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" event={"ID":"7ac60650-0f41-4ee2-9f17-22435a8897b2","Type":"ContainerDied","Data":"6886382910ecc79007db0a6af7f4750def99d623a3105e52f0af0d90a344a800"} Oct 31 03:58:44.110163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.109973 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6886382910ecc79007db0a6af7f4750def99d623a3105e52f0af0d90a344a800" Oct 31 03:58:44.110163 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.109924 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6" Oct 31 03:58:44.982805 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.982765 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8"] Oct 31 03:58:44.983444 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.983422 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7ac60650-0f41-4ee2-9f17-22435a8897b2" containerName="revoke-keys" Oct 31 03:58:44.983531 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.983448 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac60650-0f41-4ee2-9f17-22435a8897b2" containerName="revoke-keys" Oct 31 03:58:44.983592 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.983555 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac60650-0f41-4ee2-9f17-22435a8897b2" containerName="revoke-keys" Oct 31 03:58:44.986826 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.986803 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:44.993745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.993500 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:58:44.993745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.993617 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:58:44.993745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.993651 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:58:44.993745 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.993712 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:58:44.996012 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:44.995988 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8"] Oct 31 03:58:45.031813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.031759 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5pg\" (UniqueName: \"kubernetes.io/projected/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-kube-api-access-4r5pg\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.032027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.031878 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.032027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.031930 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.133478 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.133448 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5pg\" (UniqueName: \"kubernetes.io/projected/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-kube-api-access-4r5pg\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.133690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.133537 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.133690 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.133573 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.134295 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.134274 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.136164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.136138 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.141927 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.141906 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5pg\" (UniqueName: \"kubernetes.io/projected/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-kube-api-access-4r5pg\") pod \"maas-api-revoke-sub-a9c94256-b49n8\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.306608 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.306505 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:45.589911 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:45.589869 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8"] Oct 31 03:58:45.591517 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:58:45.591485 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9e2fdf_8b62_44bb_aec6_7785ff8cd80c.slice/crio-617b3e9c2cd19c4f1bdd85b25d9b4d979e695bb45d803361bff598b2f6d3332d WatchSource:0}: Error finding container 617b3e9c2cd19c4f1bdd85b25d9b4d979e695bb45d803361bff598b2f6d3332d: Status 404 returned error can't find the container with id 617b3e9c2cd19c4f1bdd85b25d9b4d979e695bb45d803361bff598b2f6d3332d Oct 31 03:58:46.130043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:46.129948 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" event={"ID":"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c","Type":"ContainerStarted","Data":"27c01bb25b864d061db7c395636a96097912f430db93f56c61b55a91b85c6b14"} Oct 31 03:58:46.130043 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:46.130008 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" event={"ID":"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c","Type":"ContainerStarted","Data":"617b3e9c2cd19c4f1bdd85b25d9b4d979e695bb45d803361bff598b2f6d3332d"} Oct 31 03:58:46.144956 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:46.144897 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" podStartSLOduration=2.1448852880000002 podStartE2EDuration="2.144885288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:46.144535359 +0000 UTC m=+2550.339058514" watchObservedRunningTime="2026-10-31 03:58:46.144885288 +0000 UTC m=+2550.339408408" Oct 31 03:58:47.136480 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:47.136442 5520 generic.go:356] "Generic (PLEG): container finished" podID="de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" containerID="27c01bb25b864d061db7c395636a96097912f430db93f56c61b55a91b85c6b14" exitCode=0 Oct 31 03:58:47.136985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:47.136519 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" event={"ID":"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c","Type":"ContainerDied","Data":"27c01bb25b864d061db7c395636a96097912f430db93f56c61b55a91b85c6b14"} Oct 31 03:58:48.298013 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.297981 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:48.364905 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.364872 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-xks-ca\") pod \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " Oct 31 03:58:48.365112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.364929 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-openshift-ca\") pod \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " Oct 31 03:58:48.365112 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.365004 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-kube-api-access-4r5pg\" (UniqueName: \"kubernetes.io/projected/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-kube-api-access-4r5pg\") pod \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\" (UID: \"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c\") " Oct 31 03:58:48.365388 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.365359 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-openshift-ca" pod "de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" (UID: "de9e2fdf-8b62-44bb-aec6-7785ff8cd80c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:48.367331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.367303 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-xks-ca" pod "de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" (UID: "de9e2fdf-8b62-44bb-aec6-7785ff8cd80c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:48.367432 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.367328 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-kube-api-access-4r5pg" pod "de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" (UID: "de9e2fdf-8b62-44bb-aec6-7785ff8cd80c"). InnerVolumeSpecName "kube-api-access-4r5pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:48.466822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.466786 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.466822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.466818 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:48.466822 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:48.466829 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5pg\" (UniqueName: \"kubernetes.io/projected/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c-kube-api-access-4r5pg\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:49.148483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.148443 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" event={"ID":"de9e2fdf-8b62-44bb-aec6-7785ff8cd80c","Type":"ContainerDied","Data":"617b3e9c2cd19c4f1bdd85b25d9b4d979e695bb45d803361bff598b2f6d3332d"} Oct 31 03:58:49.148672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.148492 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="617b3e9c2cd19c4f1bdd85b25d9b4d979e695bb45d803361bff598b2f6d3332d" Oct 31 03:58:49.148672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.148458 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8" Oct 31 03:58:49.908809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.908774 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j"] Oct 31 03:58:49.909249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.909192 5520 cpu_manager.go:414] "containerMap: removing container" podUID="de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" containerName="revoke-keys" Oct 31 03:58:49.909249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.909217 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" containerName="revoke-keys" Oct 31 03:58:49.909344 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.909280 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" containerName="revoke-keys" Oct 31 03:58:49.912667 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.912635 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:49.916159 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.916134 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-34cb76-openshift-default-dockercfg-qd8ts\"" Oct 31 03:58:49.923718 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:49.923692 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j"] Oct 31 03:58:50.000660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.000621 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.000660 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.000659 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.000926 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.000696 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgjz5\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-kube-api-access-hgjz5\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.000926 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.000836 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.001051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.001029 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.001111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.001061 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7179086-04cb-4804-9be6-842e97d9b5e2-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.001219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.001180 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.001352 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.001333 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.001429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.001415 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.102683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.102649 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.102683 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.102689 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.102939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.102715 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgjz5\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-kube-api-access-hgjz5\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.102939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.102841 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.102939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.102894 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.102939 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.102926 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7179086-04cb-4804-9be6-842e97d9b5e2-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.103242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.102974 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.103242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.103032 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.103242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.103075 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.104056 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.104021 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.104219 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.104129 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7179086-04cb-4804-9be6-842e97d9b5e2-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.107162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.104377 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.107162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.104405 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.107162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.104541 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.107162 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.105633 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.116891 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.116859 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.117818 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.117717 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.118173 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.118151 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgjz5\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-kube-api-access-hgjz5\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.228781 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.228741 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:50.508621 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.508592 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j"] Oct 31 03:58:50.511425 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:58:50.511388 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7179086_04cb_4804_9be6_842e97d9b5e2.slice/crio-220f37e3a054cccda8947c17757c1e24cd41b1666aaac03ea470128d0f328be4 WatchSource:0}: Error finding container 220f37e3a054cccda8947c17757c1e24cd41b1666aaac03ea470128d0f328be4: Status 404 returned error can't find the container with id 220f37e3a054cccda8947c17757c1e24cd41b1666aaac03ea470128d0f328be4 Oct 31 03:58:50.514381 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.514340 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:58:50.514505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.514427 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:58:50.514505 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:50.514468 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 03:58:51.161226 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:51.161156 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" event={"ID":"a7179086-04cb-4804-9be6-842e97d9b5e2","Type":"ContainerStarted","Data":"97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f"} Oct 31 03:58:51.161627 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:51.161244 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" event={"ID":"a7179086-04cb-4804-9be6-842e97d9b5e2","Type":"ContainerStarted","Data":"220f37e3a054cccda8947c17757c1e24cd41b1666aaac03ea470128d0f328be4"} Oct 31 03:58:51.229639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:51.229572 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:51.231483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:51.231456 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 03:58:51.231639 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:51.231504 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 03:58:52.229434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:52.229239 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 03:58:52.229434 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:52.229304 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 03:58:53.229903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:53.229862 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 03:58:53.230418 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:53.229920 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 03:58:54.229427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:54.229394 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 03:58:54.229618 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:54.229460 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 03:58:55.229307 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.229272 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 03:58:55.229703 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.229324 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 03:58:55.658682 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.658567 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podStartSLOduration=6.658533795 podStartE2EDuration="6.658533795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:51.188745113 +0000 UTC m=+2555.383268263" watchObservedRunningTime="2026-10-31 03:58:55.658533795 +0000 UTC m=+2559.853056916" Oct 31 03:58:55.659807 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.659785 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk"] Oct 31 03:58:55.663633 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.663614 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.681426 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.681399 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:58:55.714456 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.714410 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk"] Oct 31 03:58:55.714934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.714910 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:58:55.730182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.730149 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:58:55.730182 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.730177 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:58:55.761422 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.761389 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.761560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.761499 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.761628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.761610 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdpt\" (UniqueName: \"kubernetes.io/projected/3e0edb99-c775-4d3a-a521-56f3e339d2d7-kube-api-access-lmdpt\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.862628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.862585 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.862843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.862713 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.862843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.862815 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdpt\" (UniqueName: \"kubernetes.io/projected/3e0edb99-c775-4d3a-a521-56f3e339d2d7-kube-api-access-lmdpt\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.863340 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.863315 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.865298 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.865271 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:55.897245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:55.895117 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdpt\" (UniqueName: \"kubernetes.io/projected/3e0edb99-c775-4d3a-a521-56f3e339d2d7-kube-api-access-lmdpt\") pod \"maas-api-revoke-sub-bd145a61-fwxtk\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:56.020876 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:56.020838 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:56.229453 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:56.229422 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 31 03:58:56.229838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:56.229468 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 31 03:58:56.387291 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:56.384257 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk"] Oct 31 03:58:57.034237 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.034171 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz"] Oct 31 03:58:57.039327 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.039290 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.052985 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.051512 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz"] Oct 31 03:58:57.102756 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.102717 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.102967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.102784 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vqr\" (UniqueName: \"kubernetes.io/projected/432e6aee-6966-45ce-ab45-c72af3430abb-kube-api-access-d4vqr\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.102967 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.102814 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.204408 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.204364 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.204614 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.204457 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vqr\" (UniqueName: \"kubernetes.io/projected/432e6aee-6966-45ce-ab45-c72af3430abb-kube-api-access-d4vqr\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.204614 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.204491 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.205234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.205186 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.207310 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.207284 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.212843 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.212811 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vqr\" (UniqueName: \"kubernetes.io/projected/432e6aee-6966-45ce-ab45-c72af3430abb-kube-api-access-d4vqr\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.227893 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.227859 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" event={"ID":"3e0edb99-c775-4d3a-a521-56f3e339d2d7","Type":"ContainerStarted","Data":"44b9dc54ed9ff7253340d1e2b24370cb1aaced0b4070709b8071070e86a9ac90"} Oct 31 03:58:57.228051 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.227922 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" event={"ID":"3e0edb99-c775-4d3a-a521-56f3e339d2d7","Type":"ContainerStarted","Data":"055b637926d4041df1404f27f8b46ef38edb0ba7d20de1ab7da10aa54793b726"} Oct 31 03:58:57.243385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.243330 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" podStartSLOduration=2.243309557 podStartE2EDuration="2.243309557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:57.243165757 +0000 UTC m=+2561.437688911" watchObservedRunningTime="2026-10-31 03:58:57.243309557 +0000 UTC m=+2561.437832680" Oct 31 03:58:57.379244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.379169 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:58:57.656578 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.656550 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz"] Oct 31 03:58:57.657946 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:58:57.657923 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432e6aee_6966_45ce_ab45_c72af3430abb.slice/crio-7a47c312980d514fa6a2574419bffad69ae25c53c1b0c9ba7ff90874fce911ae WatchSource:0}: Error finding container 7a47c312980d514fa6a2574419bffad69ae25c53c1b0c9ba7ff90874fce911ae: Status 404 returned error can't find the container with id 7a47c312980d514fa6a2574419bffad69ae25c53c1b0c9ba7ff90874fce911ae Oct 31 03:58:57.711586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.711557 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:57.712345 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.712321 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:57.716484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:57.716425 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:58:58.235611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:58.235576 5520 generic.go:356] "Generic (PLEG): container finished" podID="3e0edb99-c775-4d3a-a521-56f3e339d2d7" containerID="44b9dc54ed9ff7253340d1e2b24370cb1aaced0b4070709b8071070e86a9ac90" exitCode=0 Oct 31 03:58:58.235853 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:58.235670 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" event={"ID":"3e0edb99-c775-4d3a-a521-56f3e339d2d7","Type":"ContainerDied","Data":"44b9dc54ed9ff7253340d1e2b24370cb1aaced0b4070709b8071070e86a9ac90"} Oct 31 03:58:58.237166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:58.237147 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" event={"ID":"432e6aee-6966-45ce-ab45-c72af3430abb","Type":"ContainerStarted","Data":"a5a15a05904f968ab9104c2f8661e9b463da6c3673bd9020932a722bc71ed9c5"} Oct 31 03:58:58.237267 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:58.237186 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" event={"ID":"432e6aee-6966-45ce-ab45-c72af3430abb","Type":"ContainerStarted","Data":"7a47c312980d514fa6a2574419bffad69ae25c53c1b0c9ba7ff90874fce911ae"} Oct 31 03:58:58.266469 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:58.266419 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" podStartSLOduration=1.266402446 podStartE2EDuration="1.266402446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:58:58.265886562 +0000 UTC m=+2562.460409696" watchObservedRunningTime="2026-10-31 03:58:58.266402446 +0000 UTC m=+2562.460925567" Oct 31 03:58:59.244166 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.244128 5520 generic.go:356] "Generic (PLEG): container finished" podID="432e6aee-6966-45ce-ab45-c72af3430abb" containerID="a5a15a05904f968ab9104c2f8661e9b463da6c3673bd9020932a722bc71ed9c5" exitCode=0 Oct 31 03:58:59.244431 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.244228 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" event={"ID":"432e6aee-6966-45ce-ab45-c72af3430abb","Type":"ContainerDied","Data":"a5a15a05904f968ab9104c2f8661e9b463da6c3673bd9020932a722bc71ed9c5"} Oct 31 03:58:59.416897 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.416869 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:58:59.532361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.532276 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0edb99-c775-4d3a-a521-56f3e339d2d7-kube-api-access-lmdpt\" (UniqueName: \"kubernetes.io/projected/3e0edb99-c775-4d3a-a521-56f3e339d2d7-kube-api-access-lmdpt\") pod \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " Oct 31 03:58:59.532361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.532319 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-xks-ca\") pod \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " Oct 31 03:58:59.532611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.532375 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-openshift-ca\") pod \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\" (UID: \"3e0edb99-c775-4d3a-a521-56f3e339d2d7\") " Oct 31 03:58:59.532813 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.532788 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-openshift-ca" pod "3e0edb99-c775-4d3a-a521-56f3e339d2d7" (UID: "3e0edb99-c775-4d3a-a521-56f3e339d2d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:58:59.534664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.534631 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-xks-ca" pod "3e0edb99-c775-4d3a-a521-56f3e339d2d7" (UID: "3e0edb99-c775-4d3a-a521-56f3e339d2d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:58:59.534770 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.534634 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0edb99-c775-4d3a-a521-56f3e339d2d7-kube-api-access-lmdpt" pod "3e0edb99-c775-4d3a-a521-56f3e339d2d7" (UID: "3e0edb99-c775-4d3a-a521-56f3e339d2d7"). InnerVolumeSpecName "kube-api-access-lmdpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:58:59.633724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.633687 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdpt\" (UniqueName: \"kubernetes.io/projected/3e0edb99-c775-4d3a-a521-56f3e339d2d7-kube-api-access-lmdpt\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:59.633724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.633716 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:58:59.633724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:58:59.633726 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0edb99-c775-4d3a-a521-56f3e339d2d7-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:00.251702 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.251667 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" Oct 31 03:59:00.251925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.251714 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk" event={"ID":"3e0edb99-c775-4d3a-a521-56f3e339d2d7","Type":"ContainerDied","Data":"055b637926d4041df1404f27f8b46ef38edb0ba7d20de1ab7da10aa54793b726"} Oct 31 03:59:00.251925 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.251770 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="055b637926d4041df1404f27f8b46ef38edb0ba7d20de1ab7da10aa54793b726" Oct 31 03:59:00.399257 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.399226 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:59:00.548930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.548822 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-xks-ca\") pod \"432e6aee-6966-45ce-ab45-c72af3430abb\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " Oct 31 03:59:00.548930 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.548896 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432e6aee-6966-45ce-ab45-c72af3430abb-kube-api-access-d4vqr\" (UniqueName: \"kubernetes.io/projected/432e6aee-6966-45ce-ab45-c72af3430abb-kube-api-access-d4vqr\") pod \"432e6aee-6966-45ce-ab45-c72af3430abb\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " Oct 31 03:59:00.549663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.549034 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-openshift-ca\") pod \"432e6aee-6966-45ce-ab45-c72af3430abb\" (UID: \"432e6aee-6966-45ce-ab45-c72af3430abb\") " Oct 31 03:59:00.551854 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.551813 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-openshift-ca" pod "432e6aee-6966-45ce-ab45-c72af3430abb" (UID: "432e6aee-6966-45ce-ab45-c72af3430abb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:00.553729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.553702 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:00.554533 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.554493 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432e6aee-6966-45ce-ab45-c72af3430abb-kube-api-access-d4vqr" pod "432e6aee-6966-45ce-ab45-c72af3430abb" (UID: "432e6aee-6966-45ce-ab45-c72af3430abb"). InnerVolumeSpecName "kube-api-access-d4vqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:00.561026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.560989 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-xks-ca" pod "432e6aee-6966-45ce-ab45-c72af3430abb" (UID: "432e6aee-6966-45ce-ab45-c72af3430abb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:00.655357 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.655323 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432e6aee-6966-45ce-ab45-c72af3430abb-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:00.655557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:00.655359 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vqr\" (UniqueName: \"kubernetes.io/projected/432e6aee-6966-45ce-ab45-c72af3430abb-kube-api-access-d4vqr\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:01.258921 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:01.258885 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" event={"ID":"432e6aee-6966-45ce-ab45-c72af3430abb","Type":"ContainerDied","Data":"7a47c312980d514fa6a2574419bffad69ae25c53c1b0c9ba7ff90874fce911ae"} Oct 31 03:59:01.259119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:01.258934 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a47c312980d514fa6a2574419bffad69ae25c53c1b0c9ba7ff90874fce911ae" Oct 31 03:59:01.259119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:01.258901 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz" Oct 31 03:59:06.171225 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.171179 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f"] Oct 31 03:59:06.171878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.171856 5520 cpu_manager.go:414] "containerMap: removing container" podUID="432e6aee-6966-45ce-ab45-c72af3430abb" containerName="revoke-keys" Oct 31 03:59:06.172001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.171879 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432e6aee-6966-45ce-ab45-c72af3430abb" containerName="revoke-keys" Oct 31 03:59:06.172001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.171909 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3e0edb99-c775-4d3a-a521-56f3e339d2d7" containerName="revoke-keys" Oct 31 03:59:06.172001 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.171917 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0edb99-c775-4d3a-a521-56f3e339d2d7" containerName="revoke-keys" Oct 31 03:59:06.172164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.172018 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0edb99-c775-4d3a-a521-56f3e339d2d7" containerName="revoke-keys" Oct 31 03:59:06.172164 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.172042 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="432e6aee-6966-45ce-ab45-c72af3430abb" containerName="revoke-keys" Oct 31 03:59:06.176224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.176188 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.185379 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.185310 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:06.185728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.185705 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:06.189949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.185952 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:59:06.189949 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.189629 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f"] Oct 31 03:59:06.190279 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.190235 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:06.230885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.230840 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.231245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.231194 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.231407 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.231353 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszf7\" (UniqueName: \"kubernetes.io/projected/2e43a043-741e-470f-ac12-2891ece53a6b-kube-api-access-zszf7\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.332581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.332547 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszf7\" (UniqueName: \"kubernetes.io/projected/2e43a043-741e-470f-ac12-2891ece53a6b-kube-api-access-zszf7\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.332768 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.332668 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.332768 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.332715 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.333508 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.333482 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.335597 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.335575 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.340509 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.340488 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszf7\" (UniqueName: \"kubernetes.io/projected/2e43a043-741e-470f-ac12-2891ece53a6b-kube-api-access-zszf7\") pod \"maas-api-revoke-sub-c4baf638-7rj9f\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.506242 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.506181 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:06.796752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:06.796723 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f"] Oct 31 03:59:06.799586 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:59:06.799550 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e43a043_741e_470f_ac12_2891ece53a6b.slice/crio-a31d1e2440e8e94bc34820a19565007f32ef5a480d9a39bb60b0e8fb82df394d WatchSource:0}: Error finding container a31d1e2440e8e94bc34820a19565007f32ef5a480d9a39bb60b0e8fb82df394d: Status 404 returned error can't find the container with id a31d1e2440e8e94bc34820a19565007f32ef5a480d9a39bb60b0e8fb82df394d Oct 31 03:59:07.296110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:07.296073 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" event={"ID":"2e43a043-741e-470f-ac12-2891ece53a6b","Type":"ContainerStarted","Data":"2bdf556214ef23fec943adb6ca83d73543749daa213ec42fed6b54fee2e97153"} Oct 31 03:59:07.296664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:07.296135 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" event={"ID":"2e43a043-741e-470f-ac12-2891ece53a6b","Type":"ContainerStarted","Data":"a31d1e2440e8e94bc34820a19565007f32ef5a480d9a39bb60b0e8fb82df394d"} Oct 31 03:59:07.313193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:07.313137 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" podStartSLOduration=1.313116837 podStartE2EDuration="1.313116837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:07.312455288 +0000 UTC m=+2571.506978436" watchObservedRunningTime="2026-10-31 03:59:07.313116837 +0000 UTC m=+2571.507639959" Oct 31 03:59:08.303120 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:08.303074 5520 generic.go:356] "Generic (PLEG): container finished" podID="2e43a043-741e-470f-ac12-2891ece53a6b" containerID="2bdf556214ef23fec943adb6ca83d73543749daa213ec42fed6b54fee2e97153" exitCode=0 Oct 31 03:59:08.303560 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:08.303153 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" event={"ID":"2e43a043-741e-470f-ac12-2891ece53a6b","Type":"ContainerDied","Data":"2bdf556214ef23fec943adb6ca83d73543749daa213ec42fed6b54fee2e97153"} Oct 31 03:59:08.658433 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:08.658344 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz"] Oct 31 03:59:08.663741 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:08.663708 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-20585c-xdzhz"] Oct 31 03:59:09.361475 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:09.361437 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:59:09.472405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.472380 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:09.574509 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.574425 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e43a043-741e-470f-ac12-2891ece53a6b-kube-api-access-zszf7\" (UniqueName: \"kubernetes.io/projected/2e43a043-741e-470f-ac12-2891ece53a6b-kube-api-access-zszf7\") pod \"2e43a043-741e-470f-ac12-2891ece53a6b\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " Oct 31 03:59:09.574671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.574544 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-xks-ca\") pod \"2e43a043-741e-470f-ac12-2891ece53a6b\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " Oct 31 03:59:09.574671 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.574580 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-openshift-ca\") pod \"2e43a043-741e-470f-ac12-2891ece53a6b\" (UID: \"2e43a043-741e-470f-ac12-2891ece53a6b\") " Oct 31 03:59:09.575129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.575053 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-openshift-ca" pod "2e43a043-741e-470f-ac12-2891ece53a6b" (UID: "2e43a043-741e-470f-ac12-2891ece53a6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:09.576764 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.576738 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-xks-ca" pod "2e43a043-741e-470f-ac12-2891ece53a6b" (UID: "2e43a043-741e-470f-ac12-2891ece53a6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:09.577232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.577215 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e43a043-741e-470f-ac12-2891ece53a6b-kube-api-access-zszf7" pod "2e43a043-741e-470f-ac12-2891ece53a6b" (UID: "2e43a043-741e-470f-ac12-2891ece53a6b"). InnerVolumeSpecName "kube-api-access-zszf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:09.676176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.676142 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:09.676176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.676170 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e43a043-741e-470f-ac12-2891ece53a6b-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:09.676176 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.676180 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszf7\" (UniqueName: \"kubernetes.io/projected/2e43a043-741e-470f-ac12-2891ece53a6b-kube-api-access-zszf7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:09.915961 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:09.915870 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432e6aee-6966-45ce-ab45-c72af3430abb" path="/var/lib/kubelet/pods/432e6aee-6966-45ce-ab45-c72af3430abb/volumes" Oct 31 03:59:10.315497 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:10.315467 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" Oct 31 03:59:10.315688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:10.315489 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f" event={"ID":"2e43a043-741e-470f-ac12-2891ece53a6b","Type":"ContainerDied","Data":"a31d1e2440e8e94bc34820a19565007f32ef5a480d9a39bb60b0e8fb82df394d"} Oct 31 03:59:10.315688 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:10.315551 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31d1e2440e8e94bc34820a19565007f32ef5a480d9a39bb60b0e8fb82df394d" Oct 31 03:59:15.937755 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:15.937710 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk"] Oct 31 03:59:15.938429 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:15.937940 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" containerID="cri-o://e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03" gracePeriod=30 Oct 31 03:59:16.483919 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.483881 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk"] Oct 31 03:59:16.484561 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.484542 5520 cpu_manager.go:414] "containerMap: removing container" podUID="2e43a043-741e-470f-ac12-2891ece53a6b" containerName="revoke-keys" Oct 31 03:59:16.484635 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.484565 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e43a043-741e-470f-ac12-2891ece53a6b" containerName="revoke-keys" Oct 31 03:59:16.484697 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.484685 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e43a043-741e-470f-ac12-2891ece53a6b" containerName="revoke-keys" Oct 31 03:59:16.488332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.488311 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.493026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.492303 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:16.493026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.492317 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:59:16.493026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.492356 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:16.493026 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.492407 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:16.496084 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.495934 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk"] Oct 31 03:59:16.557483 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.557452 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvlk\" (UniqueName: \"kubernetes.io/projected/adc053c7-cb26-4002-8d3e-812ed289f07f-kube-api-access-dnvlk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.557663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.557610 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.557663 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.557640 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.659117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.659075 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.659117 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.659125 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.659419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.659176 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvlk\" (UniqueName: \"kubernetes.io/projected/adc053c7-cb26-4002-8d3e-812ed289f07f-kube-api-access-dnvlk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.659854 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.659832 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.661941 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.661909 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.672308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.672283 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvlk\" (UniqueName: \"kubernetes.io/projected/adc053c7-cb26-4002-8d3e-812ed289f07f-kube-api-access-dnvlk\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:16.795485 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.795395 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g"] Oct 31 03:59:16.806581 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:16.806547 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:17.091021 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.090995 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk"] Oct 31 03:59:17.092679 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:59:17.092649 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc053c7_cb26_4002_8d3e_812ed289f07f.slice/crio-f06c9f55987c55f430597540221611dfe4270636440a7495745748312e5b20eb WatchSource:0}: Error finding container f06c9f55987c55f430597540221611dfe4270636440a7495745748312e5b20eb: Status 404 returned error can't find the container with id f06c9f55987c55f430597540221611dfe4270636440a7495745748312e5b20eb Oct 31 03:59:17.357877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.357794 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" event={"ID":"adc053c7-cb26-4002-8d3e-812ed289f07f","Type":"ContainerStarted","Data":"f06c9f55987c55f430597540221611dfe4270636440a7495745748312e5b20eb"} Oct 31 03:59:17.672186 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.672090 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz"] Oct 31 03:59:17.883875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.883839 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr"] Oct 31 03:59:17.887035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.887011 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:17.899146 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.899114 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr"] Oct 31 03:59:17.985140 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.985092 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:17.985370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.985237 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:17.985370 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:17.985290 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7m6\" (UniqueName: \"kubernetes.io/projected/adae6af6-395d-4d7e-95bd-2f4c185d0e41-kube-api-access-cl7m6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.086650 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.086623 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.086778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.086702 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.086778 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.086731 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7m6\" (UniqueName: \"kubernetes.io/projected/adae6af6-395d-4d7e-95bd-2f4c185d0e41-kube-api-access-cl7m6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.087420 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.087396 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.089002 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.088983 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.094885 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.094858 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7m6\" (UniqueName: \"kubernetes.io/projected/adae6af6-395d-4d7e-95bd-2f4c185d0e41-kube-api-access-cl7m6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.202901 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.202864 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:18.365321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.365290 5520 generic.go:356] "Generic (PLEG): container finished" podID="adc053c7-cb26-4002-8d3e-812ed289f07f" containerID="9675eed83c09d42d700e88dc7847fefb9aa497e704547e862d4dca630e6c5b4b" exitCode=0 Oct 31 03:59:18.365495 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.365341 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" event={"ID":"adc053c7-cb26-4002-8d3e-812ed289f07f","Type":"ContainerDied","Data":"9675eed83c09d42d700e88dc7847fefb9aa497e704547e862d4dca630e6c5b4b"} Oct 31 03:59:18.487878 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:18.487851 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr"] Oct 31 03:59:18.490552 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:59:18.490521 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadae6af6_395d_4d7e_95bd_2f4c185d0e41.slice/crio-e5091fc468cb1b2e4aaf38b8ed25846da34462fdda85493b463517775c783159 WatchSource:0}: Error finding container e5091fc468cb1b2e4aaf38b8ed25846da34462fdda85493b463517775c783159: Status 404 returned error can't find the container with id e5091fc468cb1b2e4aaf38b8ed25846da34462fdda85493b463517775c783159 Oct 31 03:59:19.375976 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.375940 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" event={"ID":"adae6af6-395d-4d7e-95bd-2f4c185d0e41","Type":"ContainerStarted","Data":"1e87ff301c3fe98543579b0d3d8912403a6d4daf316741d4a7133224a2b48d0b"} Oct 31 03:59:19.376452 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.375999 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" event={"ID":"adae6af6-395d-4d7e-95bd-2f4c185d0e41","Type":"ContainerStarted","Data":"e5091fc468cb1b2e4aaf38b8ed25846da34462fdda85493b463517775c783159"} Oct 31 03:59:19.394032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.393975 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" podStartSLOduration=2.393961301 podStartE2EDuration="2.393961301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:19.390856776 +0000 UTC m=+2583.585379898" watchObservedRunningTime="2026-10-31 03:59:19.393961301 +0000 UTC m=+2583.588484421" Oct 31 03:59:19.508141 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.508097 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:19.604331 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.604297 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-xks-ca\") pod \"adc053c7-cb26-4002-8d3e-812ed289f07f\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " Oct 31 03:59:19.604489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.604452 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc053c7-cb26-4002-8d3e-812ed289f07f-kube-api-access-dnvlk\" (UniqueName: \"kubernetes.io/projected/adc053c7-cb26-4002-8d3e-812ed289f07f-kube-api-access-dnvlk\") pod \"adc053c7-cb26-4002-8d3e-812ed289f07f\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " Oct 31 03:59:19.604489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.604484 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-openshift-ca\") pod \"adc053c7-cb26-4002-8d3e-812ed289f07f\" (UID: \"adc053c7-cb26-4002-8d3e-812ed289f07f\") " Oct 31 03:59:19.604838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.604809 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-openshift-ca" pod "adc053c7-cb26-4002-8d3e-812ed289f07f" (UID: "adc053c7-cb26-4002-8d3e-812ed289f07f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:19.606645 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.606619 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-xks-ca" pod "adc053c7-cb26-4002-8d3e-812ed289f07f" (UID: "adc053c7-cb26-4002-8d3e-812ed289f07f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:19.606769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.606749 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc053c7-cb26-4002-8d3e-812ed289f07f-kube-api-access-dnvlk" pod "adc053c7-cb26-4002-8d3e-812ed289f07f" (UID: "adc053c7-cb26-4002-8d3e-812ed289f07f"). InnerVolumeSpecName "kube-api-access-dnvlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:19.705877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.705845 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnvlk\" (UniqueName: \"kubernetes.io/projected/adc053c7-cb26-4002-8d3e-812ed289f07f-kube-api-access-dnvlk\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:19.705877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.705875 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:19.705877 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:19.705886 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc053c7-cb26-4002-8d3e-812ed289f07f-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:20.383077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:20.383045 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" Oct 31 03:59:20.383077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:20.383068 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk" event={"ID":"adc053c7-cb26-4002-8d3e-812ed289f07f","Type":"ContainerDied","Data":"f06c9f55987c55f430597540221611dfe4270636440a7495745748312e5b20eb"} Oct 31 03:59:20.383715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:20.383139 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f06c9f55987c55f430597540221611dfe4270636440a7495745748312e5b20eb" Oct 31 03:59:20.385111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:20.385083 5520 generic.go:356] "Generic (PLEG): container finished" podID="adae6af6-395d-4d7e-95bd-2f4c185d0e41" containerID="1e87ff301c3fe98543579b0d3d8912403a6d4daf316741d4a7133224a2b48d0b" exitCode=0 Oct 31 03:59:20.385328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:20.385152 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" event={"ID":"adae6af6-395d-4d7e-95bd-2f4c185d0e41","Type":"ContainerDied","Data":"1e87ff301c3fe98543579b0d3d8912403a6d4daf316741d4a7133224a2b48d0b"} Oct 31 03:59:21.222689 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.222625 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:59:21.326739 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.326705 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-podinfo\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.326947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.326793 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-credential-socket\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.326947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.326848 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-envoy\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.326947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.326882 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-data\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.326947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.326937 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istiod-ca-cert\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.327188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.326973 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-socket\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.327188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327029 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-kube-api-access-gx8km\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-kube-api-access-gx8km\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.327188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327072 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-certs\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.327188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327108 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-credential-socket" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.327188 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327177 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-token\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-token\") pod \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\" (UID: \"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03\") " Oct 31 03:59:21.327490 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327366 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-certs" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.327752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327540 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-socket" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.327752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327573 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istiod-ca-cert" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:21.327940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327788 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.327940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327807 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.327940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327817 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.327940 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.327826 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.328233 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.328182 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-data" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.330037 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.330016 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-token" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:21.330321 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.330303 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-envoy" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.330499 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.330482 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-kube-api-access-gx8km" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "kube-api-access-gx8km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:21.335424 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.335399 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-podinfo" pod "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" (UID: "df9ae6cd-e13b-4388-bcf0-2839ffc8bb03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:59:21.391734 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.391701 5520 generic.go:356] "Generic (PLEG): container finished" podID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerID="e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03" exitCode=0 Oct 31 03:59:21.392263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.391760 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" Oct 31 03:59:21.392263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.391781 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" event={"ID":"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03","Type":"ContainerDied","Data":"e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03"} Oct 31 03:59:21.392263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.391843 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk" event={"ID":"df9ae6cd-e13b-4388-bcf0-2839ffc8bb03","Type":"ContainerDied","Data":"85b104dca20c73bddf8dc6fd801934c0584c7fd19542d947d19dc6f4c8c1667d"} Oct 31 03:59:21.392263 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.391871 5520 scope.go:122] "RemoveContainer" containerID="e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03" Oct 31 03:59:21.405077 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.405051 5520 scope.go:122] "RemoveContainer" containerID="e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03" Oct 31 03:59:21.405432 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:21.405412 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03\": container with ID starting with e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03 not found: ID does not exist" containerID="e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03" Oct 31 03:59:21.405534 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.405439 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03"} err="failed to get container status \"e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03\": rpc error: code = NotFound desc = could not find container \"e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03\": container with ID starting with e134de366cc8a89b5e8f1d727473a8ff149c12db0f6b89a80ee82043568e7f03 not found: ID does not exist" Oct 31 03:59:21.420579 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.420551 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk"] Oct 31 03:59:21.424022 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.423997 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-l84lk"] Oct 31 03:59:21.428890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.428862 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.428890 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.428891 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.429042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.428901 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.429042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.428912 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx8km\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-kube-api-access-gx8km\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.429042 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.428921 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.535672 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.535644 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:21.630659 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.630621 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-xks-ca\") pod \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " Oct 31 03:59:21.630872 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.630702 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-openshift-ca\") pod \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " Oct 31 03:59:21.630872 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.630777 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adae6af6-395d-4d7e-95bd-2f4c185d0e41-kube-api-access-cl7m6\" (UniqueName: \"kubernetes.io/projected/adae6af6-395d-4d7e-95bd-2f4c185d0e41-kube-api-access-cl7m6\") pod \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\" (UID: \"adae6af6-395d-4d7e-95bd-2f4c185d0e41\") " Oct 31 03:59:21.631039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.631017 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-openshift-ca" pod "adae6af6-395d-4d7e-95bd-2f4c185d0e41" (UID: "adae6af6-395d-4d7e-95bd-2f4c185d0e41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:21.631193 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.631179 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.633127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.633101 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-xks-ca" pod "adae6af6-395d-4d7e-95bd-2f4c185d0e41" (UID: "adae6af6-395d-4d7e-95bd-2f4c185d0e41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:21.633239 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.633126 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adae6af6-395d-4d7e-95bd-2f4c185d0e41-kube-api-access-cl7m6" pod "adae6af6-395d-4d7e-95bd-2f4c185d0e41" (UID: "adae6af6-395d-4d7e-95bd-2f4c185d0e41"). InnerVolumeSpecName "kube-api-access-cl7m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:21.732190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.732150 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7m6\" (UniqueName: \"kubernetes.io/projected/adae6af6-395d-4d7e-95bd-2f4c185d0e41-kube-api-access-cl7m6\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.732190 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.732184 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adae6af6-395d-4d7e-95bd-2f4c185d0e41-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.918234 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:21.918131 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" path="/var/lib/kubelet/pods/df9ae6cd-e13b-4388-bcf0-2839ffc8bb03/volumes" Oct 31 03:59:22.418888 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:22.418848 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" event={"ID":"adae6af6-395d-4d7e-95bd-2f4c185d0e41","Type":"ContainerDied","Data":"e5091fc468cb1b2e4aaf38b8ed25846da34462fdda85493b463517775c783159"} Oct 31 03:59:22.419342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:22.418933 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5091fc468cb1b2e4aaf38b8ed25846da34462fdda85493b463517775c783159" Oct 31 03:59:22.419342 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:22.418932 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr" Oct 31 03:59:28.050859 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:28.050823 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp"] Oct 31 03:59:30.505509 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:30.505469 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk"] Oct 31 03:59:30.507711 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:30.507679 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-39e8cf-z27nk"] Oct 31 03:59:30.808127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:30.808041 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr"] Oct 31 03:59:30.813515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:30.813484 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-34cb76-nsrfr"] Oct 31 03:59:31.818035 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:31.817988 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" podUID="7460e080-d4f4-46c4-92c4-f1629f499cbb" containerName="main" containerID="cri-o://625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76" gracePeriod=45 Oct 31 03:59:31.917931 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:31.917895 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adae6af6-395d-4d7e-95bd-2f4c185d0e41" path="/var/lib/kubelet/pods/adae6af6-395d-4d7e-95bd-2f4c185d0e41/volumes" Oct 31 03:59:31.918336 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:31.918321 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc053c7-cb26-4002-8d3e-812ed289f07f" path="/var/lib/kubelet/pods/adc053c7-cb26-4002-8d3e-812ed289f07f/volumes" Oct 31 03:59:32.087027 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.087000 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:59:32.247355 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247317 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7460e080-d4f4-46c4-92c4-f1629f499cbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/7460e080-d4f4-46c4-92c4-f1629f499cbb-tls-certs\") pod \"7460e080-d4f4-46c4-92c4-f1629f499cbb\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " Oct 31 03:59:32.247545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247426 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7460e080-d4f4-46c4-92c4-f1629f499cbb-kube-api-access-27h8t\" (UniqueName: \"kubernetes.io/projected/7460e080-d4f4-46c4-92c4-f1629f499cbb-kube-api-access-27h8t\") pod \"7460e080-d4f4-46c4-92c4-f1629f499cbb\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " Oct 31 03:59:32.247545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247471 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-model-cache\") pod \"7460e080-d4f4-46c4-92c4-f1629f499cbb\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " Oct 31 03:59:32.247545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247513 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-tmp-dir\") pod \"7460e080-d4f4-46c4-92c4-f1629f499cbb\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " Oct 31 03:59:32.247729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247569 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-dshm\") pod \"7460e080-d4f4-46c4-92c4-f1629f499cbb\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " Oct 31 03:59:32.247729 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247620 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-home\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-home\") pod \"7460e080-d4f4-46c4-92c4-f1629f499cbb\" (UID: \"7460e080-d4f4-46c4-92c4-f1629f499cbb\") " Oct 31 03:59:32.247844 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247816 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-tmp-dir" pod "7460e080-d4f4-46c4-92c4-f1629f499cbb" (UID: "7460e080-d4f4-46c4-92c4-f1629f499cbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:32.247906 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247831 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-model-cache" pod "7460e080-d4f4-46c4-92c4-f1629f499cbb" (UID: "7460e080-d4f4-46c4-92c4-f1629f499cbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:32.247975 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.247929 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-home" pod "7460e080-d4f4-46c4-92c4-f1629f499cbb" (UID: "7460e080-d4f4-46c4-92c4-f1629f499cbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:32.248080 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.248064 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.248154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.248083 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.248154 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.248093 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.249846 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.249815 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-dshm" pod "7460e080-d4f4-46c4-92c4-f1629f499cbb" (UID: "7460e080-d4f4-46c4-92c4-f1629f499cbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:32.249966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.249854 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7460e080-d4f4-46c4-92c4-f1629f499cbb-tls-certs" pod "7460e080-d4f4-46c4-92c4-f1629f499cbb" (UID: "7460e080-d4f4-46c4-92c4-f1629f499cbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:32.249966 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.249911 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7460e080-d4f4-46c4-92c4-f1629f499cbb-kube-api-access-27h8t" pod "7460e080-d4f4-46c4-92c4-f1629f499cbb" (UID: "7460e080-d4f4-46c4-92c4-f1629f499cbb"). InnerVolumeSpecName "kube-api-access-27h8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:32.348715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.348636 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27h8t\" (UniqueName: \"kubernetes.io/projected/7460e080-d4f4-46c4-92c4-f1629f499cbb-kube-api-access-27h8t\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.348715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.348662 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7460e080-d4f4-46c4-92c4-f1629f499cbb-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.348715 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.348672 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7460e080-d4f4-46c4-92c4-f1629f499cbb-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.484585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.484543 5520 generic.go:356] "Generic (PLEG): container finished" podID="7460e080-d4f4-46c4-92c4-f1629f499cbb" containerID="625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76" exitCode=0 Oct 31 03:59:32.484585 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.484584 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" Oct 31 03:59:32.484841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.484628 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" event={"ID":"7460e080-d4f4-46c4-92c4-f1629f499cbb","Type":"ContainerDied","Data":"625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76"} Oct 31 03:59:32.484841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.484669 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g" event={"ID":"7460e080-d4f4-46c4-92c4-f1629f499cbb","Type":"ContainerDied","Data":"a7d7f09ed43927bc300ae93ba8806a52d70de48858701e28417bec234c5a3d58"} Oct 31 03:59:32.484841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.484702 5520 scope.go:122] "RemoveContainer" containerID="625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76" Oct 31 03:59:32.496389 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.496365 5520 scope.go:122] "RemoveContainer" containerID="625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76" Oct 31 03:59:32.497367 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:32.497328 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76\": container with ID starting with 625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76 not found: ID does not exist" containerID="625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76" Oct 31 03:59:32.497498 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.497376 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76"} err="failed to get container status \"625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76\": rpc error: code = NotFound desc = could not find container \"625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76\": container with ID starting with 625f5ca06d148b173d840f4615cf009b706a5c0cafdbcc78f30d7db6090b0d76 not found: ID does not exist" Oct 31 03:59:32.508380 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.508354 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g"] Oct 31 03:59:32.514687 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.514662 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-34cb76/sub-test-model-34cb76-kserve-94b4f9cff-wpv7g"] Oct 31 03:59:32.716006 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.715905 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" podUID="b538276b-f1ed-4ff6-a151-aadce56a20de" containerName="main" containerID="cri-o://01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d" gracePeriod=45 Oct 31 03:59:32.987244 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:32.987196 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:59:33.051638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.051592 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" podUID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" containerName="payload-processing" containerID="cri-o://3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198" gracePeriod=25 Oct 31 03:59:33.157938 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.157903 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b538276b-f1ed-4ff6-a151-aadce56a20de-kube-api-access-pkpzc\" (UniqueName: \"kubernetes.io/projected/b538276b-f1ed-4ff6-a151-aadce56a20de-kube-api-access-pkpzc\") pod \"b538276b-f1ed-4ff6-a151-aadce56a20de\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " Oct 31 03:59:33.158107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.157957 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-model-cache\") pod \"b538276b-f1ed-4ff6-a151-aadce56a20de\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " Oct 31 03:59:33.158107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158020 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-dshm\") pod \"b538276b-f1ed-4ff6-a151-aadce56a20de\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " Oct 31 03:59:33.158249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158128 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b538276b-f1ed-4ff6-a151-aadce56a20de-tls-certs\" (UniqueName: \"kubernetes.io/secret/b538276b-f1ed-4ff6-a151-aadce56a20de-tls-certs\") pod \"b538276b-f1ed-4ff6-a151-aadce56a20de\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " Oct 31 03:59:33.158249 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158169 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-tmp-dir\") pod \"b538276b-f1ed-4ff6-a151-aadce56a20de\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " Oct 31 03:59:33.158378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158246 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-home\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-home\") pod \"b538276b-f1ed-4ff6-a151-aadce56a20de\" (UID: \"b538276b-f1ed-4ff6-a151-aadce56a20de\") " Oct 31 03:59:33.158378 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158325 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-model-cache" pod "b538276b-f1ed-4ff6-a151-aadce56a20de" (UID: "b538276b-f1ed-4ff6-a151-aadce56a20de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:33.158515 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158488 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-tmp-dir" pod "b538276b-f1ed-4ff6-a151-aadce56a20de" (UID: "b538276b-f1ed-4ff6-a151-aadce56a20de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:33.158586 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158564 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-home" pod "b538276b-f1ed-4ff6-a151-aadce56a20de" (UID: "b538276b-f1ed-4ff6-a151-aadce56a20de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:33.158700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158677 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:33.158700 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158700 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:33.158874 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.158713 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:33.160545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.160525 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b538276b-f1ed-4ff6-a151-aadce56a20de-tls-certs" pod "b538276b-f1ed-4ff6-a151-aadce56a20de" (UID: "b538276b-f1ed-4ff6-a151-aadce56a20de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:33.160545 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.160539 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-dshm" pod "b538276b-f1ed-4ff6-a151-aadce56a20de" (UID: "b538276b-f1ed-4ff6-a151-aadce56a20de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:33.160680 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.160562 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b538276b-f1ed-4ff6-a151-aadce56a20de-kube-api-access-pkpzc" pod "b538276b-f1ed-4ff6-a151-aadce56a20de" (UID: "b538276b-f1ed-4ff6-a151-aadce56a20de"). InnerVolumeSpecName "kube-api-access-pkpzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:33.259997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.259914 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkpzc\" (UniqueName: \"kubernetes.io/projected/b538276b-f1ed-4ff6-a151-aadce56a20de-kube-api-access-pkpzc\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:33.259997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.259944 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b538276b-f1ed-4ff6-a151-aadce56a20de-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:33.259997 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.259956 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b538276b-f1ed-4ff6-a151-aadce56a20de-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:33.492006 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.491975 5520 generic.go:356] "Generic (PLEG): container finished" podID="b538276b-f1ed-4ff6-a151-aadce56a20de" containerID="01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d" exitCode=0 Oct 31 03:59:33.492248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.492040 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" Oct 31 03:59:33.492248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.492056 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" event={"ID":"b538276b-f1ed-4ff6-a151-aadce56a20de","Type":"ContainerDied","Data":"01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d"} Oct 31 03:59:33.492248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.492110 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz" event={"ID":"b538276b-f1ed-4ff6-a151-aadce56a20de","Type":"ContainerDied","Data":"71b605b782781ca6c2f4f8db4932f763b531e8de0c5ff98ed22cf0f3ea8075c8"} Oct 31 03:59:33.492248 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.492139 5520 scope.go:122] "RemoveContainer" containerID="01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d" Oct 31 03:59:33.503766 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.503740 5520 scope.go:122] "RemoveContainer" containerID="01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d" Oct 31 03:59:33.504149 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:33.504094 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d\": container with ID starting with 01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d not found: ID does not exist" containerID="01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d" Oct 31 03:59:33.504149 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.504133 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d"} err="failed to get container status \"01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d\": rpc error: code = NotFound desc = could not find container \"01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d\": container with ID starting with 01b75abd16962f95654718b1e4c0f5f7efaa0984b058108886b71b2ef6dec33d not found: ID does not exist" Oct 31 03:59:33.520272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.520171 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz"] Oct 31 03:59:33.524724 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.524694 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-ab21e4/sub-test-model-ab21e4-kserve-8487cc99-fw4dz"] Oct 31 03:59:33.915784 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.915701 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7460e080-d4f4-46c4-92c4-f1629f499cbb" path="/var/lib/kubelet/pods/7460e080-d4f4-46c4-92c4-f1629f499cbb/volumes" Oct 31 03:59:33.924368 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:33.916705 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b538276b-f1ed-4ff6-a151-aadce56a20de" path="/var/lib/kubelet/pods/b538276b-f1ed-4ff6-a151-aadce56a20de/volumes" Oct 31 03:59:38.460771 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.460735 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g"] Oct 31 03:59:38.461360 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461339 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b538276b-f1ed-4ff6-a151-aadce56a20de" containerName="main" Oct 31 03:59:38.461461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461362 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b538276b-f1ed-4ff6-a151-aadce56a20de" containerName="main" Oct 31 03:59:38.461461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461387 5520 cpu_manager.go:414] "containerMap: removing container" podUID="adae6af6-395d-4d7e-95bd-2f4c185d0e41" containerName="revoke-keys" Oct 31 03:59:38.461461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461396 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adae6af6-395d-4d7e-95bd-2f4c185d0e41" containerName="revoke-keys" Oct 31 03:59:38.461461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461407 5520 cpu_manager.go:414] "containerMap: removing container" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" Oct 31 03:59:38.461461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461415 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" Oct 31 03:59:38.461461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461449 5520 cpu_manager.go:414] "containerMap: removing container" podUID="adc053c7-cb26-4002-8d3e-812ed289f07f" containerName="revoke-keys" Oct 31 03:59:38.461461 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461456 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc053c7-cb26-4002-8d3e-812ed289f07f" containerName="revoke-keys" Oct 31 03:59:38.461835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461476 5520 cpu_manager.go:414] "containerMap: removing container" podUID="7460e080-d4f4-46c4-92c4-f1629f499cbb" containerName="main" Oct 31 03:59:38.461835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461483 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7460e080-d4f4-46c4-92c4-f1629f499cbb" containerName="main" Oct 31 03:59:38.461835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461586 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b538276b-f1ed-4ff6-a151-aadce56a20de" containerName="main" Oct 31 03:59:38.461835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461601 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="adae6af6-395d-4d7e-95bd-2f4c185d0e41" containerName="revoke-keys" Oct 31 03:59:38.461835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461610 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9ae6cd-e13b-4388-bcf0-2839ffc8bb03" containerName="istio-proxy" Oct 31 03:59:38.461835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461624 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc053c7-cb26-4002-8d3e-812ed289f07f" containerName="revoke-keys" Oct 31 03:59:38.461835 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.461633 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="7460e080-d4f4-46c4-92c4-f1629f499cbb" containerName="main" Oct 31 03:59:38.466385 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.466363 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.470119 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.470081 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:38.470443 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.470399 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:38.470562 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.470472 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:38.471308 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.471277 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:59:38.474801 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.474539 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g"] Oct 31 03:59:38.516405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.516372 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxv5\" (UniqueName: \"kubernetes.io/projected/328a4da7-1ae6-4172-a251-fc30e4b16da1-kube-api-access-qmxv5\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.516611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.516476 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.516611 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.516551 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.617275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.617234 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.617475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.617326 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.617475 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.617385 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxv5\" (UniqueName: \"kubernetes.io/projected/328a4da7-1ae6-4172-a251-fc30e4b16da1-kube-api-access-qmxv5\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.618195 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.618167 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.623125 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.620646 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.630243 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.630179 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxv5\" (UniqueName: \"kubernetes.io/projected/328a4da7-1ae6-4172-a251-fc30e4b16da1-kube-api-access-qmxv5\") pod \"maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.791746 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.791657 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:38.949630 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.949597 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j"] Oct 31 03:59:38.949837 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:38.949810 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" containerID="cri-o://97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f" gracePeriod=30 Oct 31 03:59:39.099972 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:39.099944 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g"] Oct 31 03:59:39.103178 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:59:39.103147 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328a4da7_1ae6_4172_a251_fc30e4b16da1.slice/crio-619ce5ede43e22abe559e379538ff87d99ae9f79c1b92e94567035d72f4c85f5 WatchSource:0}: Error finding container 619ce5ede43e22abe559e379538ff87d99ae9f79c1b92e94567035d72f4c85f5: Status 404 returned error can't find the container with id 619ce5ede43e22abe559e379538ff87d99ae9f79c1b92e94567035d72f4c85f5 Oct 31 03:59:39.359717 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:39.359630 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 03:59:39.532272 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:39.532233 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" event={"ID":"328a4da7-1ae6-4172-a251-fc30e4b16da1","Type":"ContainerStarted","Data":"bed900e71e71c5ff629f710a533d12efd9ed3dd33aa9af9e4958052fe764360f"} Oct 31 03:59:39.532678 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:39.532292 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" event={"ID":"328a4da7-1ae6-4172-a251-fc30e4b16da1","Type":"ContainerStarted","Data":"619ce5ede43e22abe559e379538ff87d99ae9f79c1b92e94567035d72f4c85f5"} Oct 31 03:59:39.556085 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:39.556016 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" podStartSLOduration=1.5559964339999999 podStartE2EDuration="1.555996434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:39.551063071 +0000 UTC m=+2603.745586192" watchObservedRunningTime="2026-10-31 03:59:39.555996434 +0000 UTC m=+2603.750519556" Oct 31 03:59:40.139138 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:40.139097 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" podUID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:40.539067 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:40.539021 5520 generic.go:356] "Generic (PLEG): container finished" podID="328a4da7-1ae6-4172-a251-fc30e4b16da1" containerID="bed900e71e71c5ff629f710a533d12efd9ed3dd33aa9af9e4958052fe764360f" exitCode=0 Oct 31 03:59:40.539520 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:40.539107 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" event={"ID":"328a4da7-1ae6-4172-a251-fc30e4b16da1","Type":"ContainerDied","Data":"bed900e71e71c5ff629f710a533d12efd9ed3dd33aa9af9e4958052fe764360f"} Oct 31 03:59:41.688143 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.688109 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:41.749552 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.749512 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-xks-ca\") pod \"328a4da7-1ae6-4172-a251-fc30e4b16da1\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " Oct 31 03:59:41.749751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.749576 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328a4da7-1ae6-4172-a251-fc30e4b16da1-kube-api-access-qmxv5\" (UniqueName: \"kubernetes.io/projected/328a4da7-1ae6-4172-a251-fc30e4b16da1-kube-api-access-qmxv5\") pod \"328a4da7-1ae6-4172-a251-fc30e4b16da1\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " Oct 31 03:59:41.749751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.749684 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-openshift-ca\") pod \"328a4da7-1ae6-4172-a251-fc30e4b16da1\" (UID: \"328a4da7-1ae6-4172-a251-fc30e4b16da1\") " Oct 31 03:59:41.750114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.750082 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-openshift-ca" pod "328a4da7-1ae6-4172-a251-fc30e4b16da1" (UID: "328a4da7-1ae6-4172-a251-fc30e4b16da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:41.751838 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.751809 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328a4da7-1ae6-4172-a251-fc30e4b16da1-kube-api-access-qmxv5" pod "328a4da7-1ae6-4172-a251-fc30e4b16da1" (UID: "328a4da7-1ae6-4172-a251-fc30e4b16da1"). InnerVolumeSpecName "kube-api-access-qmxv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:41.751934 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.751879 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-xks-ca" pod "328a4da7-1ae6-4172-a251-fc30e4b16da1" (UID: "328a4da7-1ae6-4172-a251-fc30e4b16da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:41.851610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.851521 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:41.851610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.851552 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxv5\" (UniqueName: \"kubernetes.io/projected/328a4da7-1ae6-4172-a251-fc30e4b16da1-kube-api-access-qmxv5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:41.851610 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:41.851562 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328a4da7-1ae6-4172-a251-fc30e4b16da1-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:42.551628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:42.551599 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" Oct 31 03:59:42.551628 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:42.551619 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g" event={"ID":"328a4da7-1ae6-4172-a251-fc30e4b16da1","Type":"ContainerDied","Data":"619ce5ede43e22abe559e379538ff87d99ae9f79c1b92e94567035d72f4c85f5"} Oct 31 03:59:42.551873 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:42.551673 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619ce5ede43e22abe559e379538ff87d99ae9f79c1b92e94567035d72f4c85f5" Oct 31 03:59:42.714875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:42.714826 5520 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:59:42.715282 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:42.714904 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:59:43.264050 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:43.264012 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz"] Oct 31 03:59:43.281223 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:43.281179 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh"] Oct 31 03:59:43.296045 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:43.296013 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n"] Oct 31 03:59:44.203139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.203112 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:59:44.279707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.279631 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-envoy\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.279707 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.279703 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-socket\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.279951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.279727 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-certs\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.279951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.279757 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-token\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-token\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.279951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.279790 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-credential-socket\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.279951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.279817 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-podinfo\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.279951 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.279907 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-kube-api-access-hgjz5\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-kube-api-access-hgjz5\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.280265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280056 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-certs" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.280265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280065 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-socket" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.280265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280140 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7179086-04cb-4804-9be6-842e97d9b5e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7179086-04cb-4804-9be6-842e97d9b5e2-istiod-ca-cert\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.280265 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280193 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-data\") pod \"a7179086-04cb-4804-9be6-842e97d9b5e2\" (UID: \"a7179086-04cb-4804-9be6-842e97d9b5e2\") " Oct 31 03:59:44.280695 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280642 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.280763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280632 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-data" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.280763 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280701 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7179086-04cb-4804-9be6-842e97d9b5e2-istiod-ca-cert" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:44.280886 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280752 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.280950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.280897 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-credential-socket" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.282271 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.282192 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-envoy" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.282392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.282297 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-kube-api-access-hgjz5" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "kube-api-access-hgjz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:44.282392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.282347 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-token" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:44.282392 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.282352 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-podinfo" pod "a7179086-04cb-4804-9be6-842e97d9b5e2" (UID: "a7179086-04cb-4804-9be6-842e97d9b5e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:59:44.382224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.382171 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgjz5\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-kube-api-access-hgjz5\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.382224 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.382229 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7179086-04cb-4804-9be6-842e97d9b5e2-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.382447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.382241 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.382447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.382250 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.382447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.382259 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.382447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.382267 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7179086-04cb-4804-9be6-842e97d9b5e2-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.382447 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.382275 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7179086-04cb-4804-9be6-842e97d9b5e2-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.563529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.563441 5520 generic.go:356] "Generic (PLEG): container finished" podID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerID="97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f" exitCode=0 Oct 31 03:59:44.563529 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.563504 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" Oct 31 03:59:44.563730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.563523 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" event={"ID":"a7179086-04cb-4804-9be6-842e97d9b5e2","Type":"ContainerDied","Data":"97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f"} Oct 31 03:59:44.563730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.563601 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j" event={"ID":"a7179086-04cb-4804-9be6-842e97d9b5e2","Type":"ContainerDied","Data":"220f37e3a054cccda8947c17757c1e24cd41b1666aaac03ea470128d0f328be4"} Oct 31 03:59:44.563730 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.563623 5520 scope.go:122] "RemoveContainer" containerID="97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f" Oct 31 03:59:44.574241 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.574192 5520 scope.go:122] "RemoveContainer" containerID="97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f" Oct 31 03:59:44.574524 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:44.574495 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f\": container with ID starting with 97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f not found: ID does not exist" containerID="97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f" Oct 31 03:59:44.574601 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.574535 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f"} err="failed to get container status \"97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f\": rpc error: code = NotFound desc = could not find container \"97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f\": container with ID starting with 97661bd2122e80d874c57afa910b7b0db9694ebe4caa26694745c2d6ffe8265f not found: ID does not exist" Oct 31 03:59:44.587110 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.587082 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j"] Oct 31 03:59:44.588809 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:44.588785 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-b7z8j"] Oct 31 03:59:45.915811 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:45.915773 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" path="/var/lib/kubelet/pods/a7179086-04cb-4804-9be6-842e97d9b5e2/volumes" Oct 31 03:59:49.610111 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:49.610080 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9"] Oct 31 03:59:50.138907 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.138869 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" podUID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:50.450232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.450180 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh"] Oct 31 03:59:50.450607 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.450595 5520 cpu_manager.go:414] "containerMap: removing container" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" Oct 31 03:59:50.450657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.450608 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" Oct 31 03:59:50.450657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.450628 5520 cpu_manager.go:414] "containerMap: removing container" podUID="328a4da7-1ae6-4172-a251-fc30e4b16da1" containerName="revoke-keys" Oct 31 03:59:50.450657 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.450634 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328a4da7-1ae6-4172-a251-fc30e4b16da1" containerName="revoke-keys" Oct 31 03:59:50.450780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.450712 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7179086-04cb-4804-9be6-842e97d9b5e2" containerName="istio-proxy" Oct 31 03:59:50.450780 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.450753 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="328a4da7-1ae6-4172-a251-fc30e4b16da1" containerName="revoke-keys" Oct 31 03:59:50.456903 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.456868 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.462800 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.462774 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:50.464127 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.463979 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:50.464554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.464315 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:50.464554 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.464443 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:59:50.464831 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.464808 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh"] Oct 31 03:59:50.549041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.548997 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.549041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.549037 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.549301 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.549113 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4dn\" (UniqueName: \"kubernetes.io/projected/6f01cb39-fe54-46d5-a39e-6e379548da76-kube-api-access-hq4dn\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.650623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.650584 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.650623 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.650623 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.651094 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.650645 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4dn\" (UniqueName: \"kubernetes.io/projected/6f01cb39-fe54-46d5-a39e-6e379548da76-kube-api-access-hq4dn\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.651419 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.651389 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.653211 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.653178 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.658752 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.658718 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4dn\" (UniqueName: \"kubernetes.io/projected/6f01cb39-fe54-46d5-a39e-6e379548da76-kube-api-access-hq4dn\") pod \"maas-api-revoke-sub-915e8093-ff5hh\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:50.779503 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:50.779397 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:51.066735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:51.066695 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh"] Oct 31 03:59:51.069616 ip-10-0-141-153 kubenswrapper[5520]: W1031 03:59:51.069581 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f01cb39_fe54_46d5_a39e_6e379548da76.slice/crio-e73ac6f0b3bddfef0427ec3d135a03e30147279c14320fe3c17eb95eb6b3bc75 WatchSource:0}: Error finding container e73ac6f0b3bddfef0427ec3d135a03e30147279c14320fe3c17eb95eb6b3bc75: Status 404 returned error can't find the container with id e73ac6f0b3bddfef0427ec3d135a03e30147279c14320fe3c17eb95eb6b3bc75 Oct 31 03:59:51.606875 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:51.606814 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" event={"ID":"6f01cb39-fe54-46d5-a39e-6e379548da76","Type":"ContainerStarted","Data":"a4e724710cf2f56ab0b4ea6140e4b2cbbf5d9a451f2d0d40dc2253f06ae13a56"} Oct 31 03:59:51.607098 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:51.606904 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" event={"ID":"6f01cb39-fe54-46d5-a39e-6e379548da76","Type":"ContainerStarted","Data":"e73ac6f0b3bddfef0427ec3d135a03e30147279c14320fe3c17eb95eb6b3bc75"} Oct 31 03:59:51.625427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:51.625299 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" podStartSLOduration=1.6252840549999998 podStartE2EDuration="1.625284055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:51.623518515 +0000 UTC m=+2615.818041664" watchObservedRunningTime="2026-10-31 03:59:51.625284055 +0000 UTC m=+2615.819807176" Oct 31 03:59:52.619637 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:52.619599 5520 generic.go:356] "Generic (PLEG): container finished" podID="6f01cb39-fe54-46d5-a39e-6e379548da76" containerID="a4e724710cf2f56ab0b4ea6140e4b2cbbf5d9a451f2d0d40dc2253f06ae13a56" exitCode=0 Oct 31 03:59:52.620088 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:52.619684 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" event={"ID":"6f01cb39-fe54-46d5-a39e-6e379548da76","Type":"ContainerDied","Data":"a4e724710cf2f56ab0b4ea6140e4b2cbbf5d9a451f2d0d40dc2253f06ae13a56"} Oct 31 03:59:53.316349 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.316283 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:59:53.377057 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.377023 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f034f9-48d8-4626-a8e0-a0ff289a3adf-kube-api-access-c9gg4\" (UniqueName: \"kubernetes.io/projected/28f034f9-48d8-4626-a8e0-a0ff289a3adf-kube-api-access-c9gg4\") pod \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " Oct 31 03:59:53.377281 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.377133 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f034f9-48d8-4626-a8e0-a0ff289a3adf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f034f9-48d8-4626-a8e0-a0ff289a3adf-plugins-config-volume\") pod \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\" (UID: \"28f034f9-48d8-4626-a8e0-a0ff289a3adf\") " Oct 31 03:59:53.377829 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.377766 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f034f9-48d8-4626-a8e0-a0ff289a3adf-plugins-config-volume" pod "28f034f9-48d8-4626-a8e0-a0ff289a3adf" (UID: "28f034f9-48d8-4626-a8e0-a0ff289a3adf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:53.379686 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.379628 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f034f9-48d8-4626-a8e0-a0ff289a3adf-kube-api-access-c9gg4" pod "28f034f9-48d8-4626-a8e0-a0ff289a3adf" (UID: "28f034f9-48d8-4626-a8e0-a0ff289a3adf"). InnerVolumeSpecName "kube-api-access-c9gg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:53.478774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.478734 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9gg4\" (UniqueName: \"kubernetes.io/projected/28f034f9-48d8-4626-a8e0-a0ff289a3adf-kube-api-access-c9gg4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:53.478774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.478777 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f034f9-48d8-4626-a8e0-a0ff289a3adf-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:53.626774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.626687 5520 generic.go:356] "Generic (PLEG): container finished" podID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" containerID="3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198" exitCode=0 Oct 31 03:59:53.626774 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.626743 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" Oct 31 03:59:53.627325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.626778 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" event={"ID":"28f034f9-48d8-4626-a8e0-a0ff289a3adf","Type":"ContainerDied","Data":"3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198"} Oct 31 03:59:53.627325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.626837 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp" event={"ID":"28f034f9-48d8-4626-a8e0-a0ff289a3adf","Type":"ContainerDied","Data":"5d5d6597b1dffd397e8553376a1db8dd88e9dc52675c148658049e25ce53417f"} Oct 31 03:59:53.627325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.626860 5520 scope.go:122] "RemoveContainer" containerID="3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198" Oct 31 03:59:53.636950 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.636927 5520 scope.go:122] "RemoveContainer" containerID="3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198" Oct 31 03:59:53.637310 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:53.637280 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198\": container with ID starting with 3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198 not found: ID does not exist" containerID="3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198" Oct 31 03:59:53.637375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.637323 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198"} err="failed to get container status \"3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198\": rpc error: code = NotFound desc = could not find container \"3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198\": container with ID starting with 3b0ad57424ba3b4b2d1b51c3030b0d5958ff1e12c1ba3ff5609276fbfd21e198 not found: ID does not exist" Oct 31 03:59:53.650253 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.650194 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp"] Oct 31 03:59:53.652375 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.652351 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-34cb76-78fdc8b8f-8c2lp"] Oct 31 03:59:53.776018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.775994 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:53.883018 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.882932 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f01cb39-fe54-46d5-a39e-6e379548da76-kube-api-access-hq4dn\" (UniqueName: \"kubernetes.io/projected/6f01cb39-fe54-46d5-a39e-6e379548da76-kube-api-access-hq4dn\") pod \"6f01cb39-fe54-46d5-a39e-6e379548da76\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " Oct 31 03:59:53.883229 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.883152 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-openshift-ca\") pod \"6f01cb39-fe54-46d5-a39e-6e379548da76\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " Oct 31 03:59:53.883325 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.883306 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-xks-ca\") pod \"6f01cb39-fe54-46d5-a39e-6e379548da76\" (UID: \"6f01cb39-fe54-46d5-a39e-6e379548da76\") " Oct 31 03:59:53.883484 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.883438 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-openshift-ca" pod "6f01cb39-fe54-46d5-a39e-6e379548da76" (UID: "6f01cb39-fe54-46d5-a39e-6e379548da76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:53.883760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.883727 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:53.885518 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.885491 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-xks-ca" pod "6f01cb39-fe54-46d5-a39e-6e379548da76" (UID: "6f01cb39-fe54-46d5-a39e-6e379548da76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:53.886102 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.886081 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f01cb39-fe54-46d5-a39e-6e379548da76-kube-api-access-hq4dn" pod "6f01cb39-fe54-46d5-a39e-6e379548da76" (UID: "6f01cb39-fe54-46d5-a39e-6e379548da76"). InnerVolumeSpecName "kube-api-access-hq4dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:53.916536 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.916501 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" path="/var/lib/kubelet/pods/28f034f9-48d8-4626-a8e0-a0ff289a3adf/volumes" Oct 31 03:59:53.995923 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.995847 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq4dn\" (UniqueName: \"kubernetes.io/projected/6f01cb39-fe54-46d5-a39e-6e379548da76-kube-api-access-hq4dn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:53.995923 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:53.995893 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f01cb39-fe54-46d5-a39e-6e379548da76-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:54.611361 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:54.611316 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" podUID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" containerName="payload-pre-processing" containerID="cri-o://b93e9b6e343de1f1b1d0a7e3de512664410f574b65880bd768a4596cc4a8d7fa" gracePeriod=25 Oct 31 03:59:54.637996 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:54.637963 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" Oct 31 03:59:54.638500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:54.637956 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh" event={"ID":"6f01cb39-fe54-46d5-a39e-6e379548da76","Type":"ContainerDied","Data":"e73ac6f0b3bddfef0427ec3d135a03e30147279c14320fe3c17eb95eb6b3bc75"} Oct 31 03:59:54.638500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:54.638114 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e73ac6f0b3bddfef0427ec3d135a03e30147279c14320fe3c17eb95eb6b3bc75" Oct 31 03:59:55.072090 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:55.072051 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" podUID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:58.286151 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.286103 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" containerID="cri-o://40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676" gracePeriod=45 Oct 31 03:59:58.306332 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.306113 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" containerID="cri-o://e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e" gracePeriod=45 Oct 31 03:59:58.319699 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.319658 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" containerID="cri-o://1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d" gracePeriod=45 Oct 31 03:59:58.323139 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.323106 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused" Oct 31 03:59:58.336258 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.336222 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused" Oct 31 03:59:58.634116 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.634086 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:59:58.638394 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.638368 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:59:58.642405 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.642365 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:59:58.670039 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.670002 5520 generic.go:356] "Generic (PLEG): container finished" podID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerID="40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676" exitCode=0 Oct 31 03:59:58.670238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.670061 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" Oct 31 03:59:58.670238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.670076 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" event={"ID":"151ddd1e-4ecc-4bb7-80f6-e0b630375cab","Type":"ContainerDied","Data":"40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676"} Oct 31 03:59:58.670238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.670121 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz" event={"ID":"151ddd1e-4ecc-4bb7-80f6-e0b630375cab","Type":"ContainerDied","Data":"3e858075b5d54f0ac605b37cccfcdb800f21795fcfebc68db629c33c993d2386"} Oct 31 03:59:58.670238 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.670141 5520 scope.go:122] "RemoveContainer" containerID="40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676" Oct 31 03:59:58.674314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.674017 5520 generic.go:356] "Generic (PLEG): container finished" podID="b391e679-e308-499e-8b85-022626eeecde" containerID="1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d" exitCode=0 Oct 31 03:59:58.674314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.674147 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" event={"ID":"b391e679-e308-499e-8b85-022626eeecde","Type":"ContainerDied","Data":"1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d"} Oct 31 03:59:58.674314 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.674182 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" event={"ID":"b391e679-e308-499e-8b85-022626eeecde","Type":"ContainerDied","Data":"dbe03aca846e3f0261574d3d52f10ae38674a55918889574375cad1a14f70d32"} Oct 31 03:59:58.674841 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.674815 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n" Oct 31 03:59:58.677664 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.677635 5520 generic.go:356] "Generic (PLEG): container finished" podID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerID="e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e" exitCode=0 Oct 31 03:59:58.677798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.677711 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" event={"ID":"8b2833b3-ae04-427f-812c-ed7174eb6d6a","Type":"ContainerDied","Data":"e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e"} Oct 31 03:59:58.677798 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.677740 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" event={"ID":"8b2833b3-ae04-427f-812c-ed7174eb6d6a","Type":"ContainerDied","Data":"a85e37017213e0e8da81625c4ebe2a6f56ddfeec12fb0354377b983deda5dd18"} Oct 31 03:59:58.677882 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.677804 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh" Oct 31 03:59:58.688760 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.688734 5520 scope.go:122] "RemoveContainer" containerID="40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676" Oct 31 03:59:58.689162 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:58.689133 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676\": container with ID starting with 40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676 not found: ID does not exist" containerID="40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676" Oct 31 03:59:58.689328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.689177 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676"} err="failed to get container status \"40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676\": rpc error: code = NotFound desc = could not find container \"40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676\": container with ID starting with 40f2e983cb1883448bf1cbea77c91c086fe4f0d316508fecccaf2e0566cc1676 not found: ID does not exist" Oct 31 03:59:58.689328 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.689219 5520 scope.go:122] "RemoveContainer" containerID="1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d" Oct 31 03:59:58.700245 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.700188 5520 scope.go:122] "RemoveContainer" containerID="1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d" Oct 31 03:59:58.700596 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:58.700572 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d\": container with ID starting with 1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d not found: ID does not exist" containerID="1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d" Oct 31 03:59:58.700705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.700608 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d"} err="failed to get container status \"1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d\": rpc error: code = NotFound desc = could not find container \"1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d\": container with ID starting with 1a56fe2ea85bdce9d946e53b62e61f11241254d1a1337d492d8739cf0f8a994d not found: ID does not exist" Oct 31 03:59:58.700705 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.700633 5520 scope.go:122] "RemoveContainer" containerID="e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e" Oct 31 03:59:58.713032 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.713005 5520 scope.go:122] "RemoveContainer" containerID="e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e" Oct 31 03:59:58.713477 ip-10-0-141-153 kubenswrapper[5520]: E1031 03:59:58.713437 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e\": container with ID starting with e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e not found: ID does not exist" containerID="e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e" Oct 31 03:59:58.713598 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.713483 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e"} err="failed to get container status \"e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e\": rpc error: code = NotFound desc = could not find container \"e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e\": container with ID starting with e25f1d6eaf25c12234b5615ebf62fa418bbbebcbec23836fec984cf50e20f20e not found: ID does not exist" Oct 31 03:59:58.747557 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747528 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tmp-dir\") pod \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " Oct 31 03:59:58.747728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747573 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-dshm\") pod \"b391e679-e308-499e-8b85-022626eeecde\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " Oct 31 03:59:58.747728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747605 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-kube-api-access-knq7q\" (UniqueName: \"kubernetes.io/projected/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-kube-api-access-knq7q\") pod \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " Oct 31 03:59:58.747728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747639 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-home\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-home\") pod \"b391e679-e308-499e-8b85-022626eeecde\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " Oct 31 03:59:58.747728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747679 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-dshm\") pod \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " Oct 31 03:59:58.747728 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747724 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tls-certs\") pod \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " Oct 31 03:59:58.748030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747783 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-model-cache\") pod \"b391e679-e308-499e-8b85-022626eeecde\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " Oct 31 03:59:58.748030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747823 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b391e679-e308-499e-8b85-022626eeecde-kube-api-access-sxxgd\" (UniqueName: \"kubernetes.io/projected/b391e679-e308-499e-8b85-022626eeecde-kube-api-access-sxxgd\") pod \"b391e679-e308-499e-8b85-022626eeecde\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " Oct 31 03:59:58.748030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747857 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2833b3-ae04-427f-812c-ed7174eb6d6a-kube-api-access-2c57d\" (UniqueName: \"kubernetes.io/projected/8b2833b3-ae04-427f-812c-ed7174eb6d6a-kube-api-access-2c57d\") pod \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " Oct 31 03:59:58.748030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747858 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tmp-dir" pod "151ddd1e-4ecc-4bb7-80f6-e0b630375cab" (UID: "151ddd1e-4ecc-4bb7-80f6-e0b630375cab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.748030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747890 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tmp-dir\") pod \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " Oct 31 03:59:58.748030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747948 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-home" pod "b391e679-e308-499e-8b85-022626eeecde" (UID: "b391e679-e308-499e-8b85-022626eeecde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.748030 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.747952 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-home\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-home\") pod \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748053 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-dshm\") pod \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748067 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tmp-dir" pod "8b2833b3-ae04-427f-812c-ed7174eb6d6a" (UID: "8b2833b3-ae04-427f-812c-ed7174eb6d6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748088 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-model-cache\") pod \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748129 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-model-cache\") pod \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748164 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-home\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-home\") pod \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\" (UID: \"8b2833b3-ae04-427f-812c-ed7174eb6d6a\") " Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748219 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b391e679-e308-499e-8b85-022626eeecde-tls-certs\" (UniqueName: \"kubernetes.io/secret/b391e679-e308-499e-8b85-022626eeecde-tls-certs\") pod \"b391e679-e308-499e-8b85-022626eeecde\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748249 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tls-certs\" (UniqueName: \"kubernetes.io/secret/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tls-certs\") pod \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\" (UID: \"151ddd1e-4ecc-4bb7-80f6-e0b630375cab\") " Oct 31 03:59:58.748470 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748309 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-tmp-dir\") pod \"b391e679-e308-499e-8b85-022626eeecde\" (UID: \"b391e679-e308-499e-8b85-022626eeecde\") " Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748130 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-home" pod "151ddd1e-4ecc-4bb7-80f6-e0b630375cab" (UID: "151ddd1e-4ecc-4bb7-80f6-e0b630375cab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748347 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-model-cache" pod "151ddd1e-4ecc-4bb7-80f6-e0b630375cab" (UID: "151ddd1e-4ecc-4bb7-80f6-e0b630375cab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748535 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-home" pod "8b2833b3-ae04-427f-812c-ed7174eb6d6a" (UID: "8b2833b3-ae04-427f-812c-ed7174eb6d6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748672 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748688 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-model-cache" pod "8b2833b3-ae04-427f-812c-ed7174eb6d6a" (UID: "8b2833b3-ae04-427f-812c-ed7174eb6d6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748692 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748729 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748747 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748762 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.748904 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.748776 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.749473 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.749456 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-tmp-dir" pod "b391e679-e308-499e-8b85-022626eeecde" (UID: "b391e679-e308-499e-8b85-022626eeecde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.750066 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.750036 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-model-cache" pod "b391e679-e308-499e-8b85-022626eeecde" (UID: "b391e679-e308-499e-8b85-022626eeecde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.750790 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.750763 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-kube-api-access-knq7q" pod "151ddd1e-4ecc-4bb7-80f6-e0b630375cab" (UID: "151ddd1e-4ecc-4bb7-80f6-e0b630375cab"). InnerVolumeSpecName "kube-api-access-knq7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:58.751107 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.751083 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-dshm" pod "8b2833b3-ae04-427f-812c-ed7174eb6d6a" (UID: "8b2833b3-ae04-427f-812c-ed7174eb6d6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.751275 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.751242 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-dshm" pod "b391e679-e308-499e-8b85-022626eeecde" (UID: "b391e679-e308-499e-8b85-022626eeecde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.751489 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.751467 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2833b3-ae04-427f-812c-ed7174eb6d6a-kube-api-access-2c57d" pod "8b2833b3-ae04-427f-812c-ed7174eb6d6a" (UID: "8b2833b3-ae04-427f-812c-ed7174eb6d6a"). InnerVolumeSpecName "kube-api-access-2c57d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:58.751915 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.751890 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b391e679-e308-499e-8b85-022626eeecde-kube-api-access-sxxgd" pod "b391e679-e308-499e-8b85-022626eeecde" (UID: "b391e679-e308-499e-8b85-022626eeecde"). InnerVolumeSpecName "kube-api-access-sxxgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:58.752129 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.752107 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tls-certs" pod "151ddd1e-4ecc-4bb7-80f6-e0b630375cab" (UID: "151ddd1e-4ecc-4bb7-80f6-e0b630375cab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:58.752947 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.752928 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b391e679-e308-499e-8b85-022626eeecde-tls-certs" pod "b391e679-e308-499e-8b85-022626eeecde" (UID: "b391e679-e308-499e-8b85-022626eeecde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:58.753023 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.753005 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-dshm" pod "151ddd1e-4ecc-4bb7-80f6-e0b630375cab" (UID: "151ddd1e-4ecc-4bb7-80f6-e0b630375cab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:58.753099 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.753082 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tls-certs" pod "8b2833b3-ae04-427f-812c-ed7174eb6d6a" (UID: "8b2833b3-ae04-427f-812c-ed7174eb6d6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:58.849735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849643 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.849735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849677 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b391e679-e308-499e-8b85-022626eeecde-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.849735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849690 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.849735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849702 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.849735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849714 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.849735 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849726 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knq7q\" (UniqueName: \"kubernetes.io/projected/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-kube-api-access-knq7q\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.850074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849742 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/151ddd1e-4ecc-4bb7-80f6-e0b630375cab-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.850074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849754 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2833b3-ae04-427f-812c-ed7174eb6d6a-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.850074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849764 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b391e679-e308-499e-8b85-022626eeecde-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.850074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849778 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxxgd\" (UniqueName: \"kubernetes.io/projected/b391e679-e308-499e-8b85-022626eeecde-kube-api-access-sxxgd\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.850074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849791 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c57d\" (UniqueName: \"kubernetes.io/projected/8b2833b3-ae04-427f-812c-ed7174eb6d6a-kube-api-access-2c57d\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.850074 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.849802 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2833b3-ae04-427f-812c-ed7174eb6d6a-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:58.996041 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.996015 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz"] Oct 31 03:59:58.999857 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:58.999831 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-2-w6-7f3159-kserve-657cfbbff5-2rhcz"] Oct 31 03:59:59.010356 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.010332 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh"] Oct 31 03:59:59.016769 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.016740 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/premium-w6-7f3159-kserve-8544497f85-bdgrh"] Oct 31 03:59:59.026354 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.026330 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n"] Oct 31 03:59:59.032114 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.032090 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/simulator-w6-7f3159-kserve-6b696459b4-b5n9n"] Oct 31 03:59:59.679719 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.679689 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5"] Oct 31 03:59:59.680335 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680315 5520 cpu_manager.go:414] "containerMap: removing container" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" Oct 31 03:59:59.680427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680337 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" Oct 31 03:59:59.680427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680375 5520 cpu_manager.go:414] "containerMap: removing container" podUID="6f01cb39-fe54-46d5-a39e-6e379548da76" containerName="revoke-keys" Oct 31 03:59:59.680427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680383 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f01cb39-fe54-46d5-a39e-6e379548da76" containerName="revoke-keys" Oct 31 03:59:59.680427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680406 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" Oct 31 03:59:59.680427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680413 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" Oct 31 03:59:59.680427 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680427 5520 cpu_manager.go:414] "containerMap: removing container" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680434 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680446 5520 cpu_manager.go:414] "containerMap: removing container" podUID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" containerName="payload-processing" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680453 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" containerName="payload-processing" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680540 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" containerName="main" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680557 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f01cb39-fe54-46d5-a39e-6e379548da76" containerName="revoke-keys" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680571 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b391e679-e308-499e-8b85-022626eeecde" containerName="main" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680581 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" containerName="main" Oct 31 03:59:59.680748 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.680590 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f034f9-48d8-4626-a8e0-a0ff289a3adf" containerName="payload-processing" Oct 31 03:59:59.694034 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.694005 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.695779 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.695752 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5"] Oct 31 03:59:59.697638 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.697408 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 03:59:59.698591 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.698399 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 03:59:59.698591 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.698451 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:59:59.698591 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.698523 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:59:59.858986 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.858949 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgmc\" (UniqueName: \"kubernetes.io/projected/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-kube-api-access-jsgmc\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.859232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.859018 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.859232 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.859064 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.915817 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.915782 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151ddd1e-4ecc-4bb7-80f6-e0b630375cab" path="/var/lib/kubelet/pods/151ddd1e-4ecc-4bb7-80f6-e0b630375cab/volumes" Oct 31 03:59:59.916268 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.916253 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2833b3-ae04-427f-812c-ed7174eb6d6a" path="/var/lib/kubelet/pods/8b2833b3-ae04-427f-812c-ed7174eb6d6a/volumes" Oct 31 03:59:59.916652 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.916637 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b391e679-e308-499e-8b85-022626eeecde" path="/var/lib/kubelet/pods/b391e679-e308-499e-8b85-022626eeecde/volumes" Oct 31 03:59:59.959803 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.959726 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.959803 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.959766 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.960061 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.959845 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgmc\" (UniqueName: \"kubernetes.io/projected/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-kube-api-access-jsgmc\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.960535 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.960510 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.962500 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.962476 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 03:59:59.967751 ip-10-0-141-153 kubenswrapper[5520]: I1031 03:59:59.967709 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgmc\" (UniqueName: \"kubernetes.io/projected/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-kube-api-access-jsgmc\") pod \"maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 04:00:00.009946 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.009905 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 04:00:00.149640 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.149597 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64"] Oct 31 04:00:00.154930 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.154908 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.164668 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.162559 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64"] Oct 31 04:00:00.262895 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.262809 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7zl\" (UniqueName: \"kubernetes.io/projected/bc1aacc1-81da-4014-8405-aaabadd1a281-kube-api-access-9r7zl\") pod \"maas-api-key-cleanup-29890320-cwh64\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.262895 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.262857 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1aacc1-81da-4014-8405-aaabadd1a281-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-cwh64\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.307959 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.307933 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5"] Oct 31 04:00:00.309545 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:00:00.309513 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f8b9be_b4aa_490a_8c32_5c89c913f8c8.slice/crio-83ede8833f1dbc9a3df519a7a256548e42b5bf08d5301fa1c0978348e03e89dd WatchSource:0}: Error finding container 83ede8833f1dbc9a3df519a7a256548e42b5bf08d5301fa1c0978348e03e89dd: Status 404 returned error can't find the container with id 83ede8833f1dbc9a3df519a7a256548e42b5bf08d5301fa1c0978348e03e89dd Oct 31 04:00:00.364445 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.364408 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7zl\" (UniqueName: \"kubernetes.io/projected/bc1aacc1-81da-4014-8405-aaabadd1a281-kube-api-access-9r7zl\") pod \"maas-api-key-cleanup-29890320-cwh64\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.364587 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.364460 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1aacc1-81da-4014-8405-aaabadd1a281-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-cwh64\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.365071 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.365052 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1aacc1-81da-4014-8405-aaabadd1a281-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-cwh64\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.388304 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.388266 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7zl\" (UniqueName: \"kubernetes.io/projected/bc1aacc1-81da-4014-8405-aaabadd1a281-kube-api-access-9r7zl\") pod \"maas-api-key-cleanup-29890320-cwh64\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.474965 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.474932 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:00.708024 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.707342 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" event={"ID":"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8","Type":"ContainerStarted","Data":"051a8f2c0b4afadadddd0804c34056893852fa62a69d737918fc8ff5387f8761"} Oct 31 04:00:00.708024 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.707407 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" event={"ID":"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8","Type":"ContainerStarted","Data":"83ede8833f1dbc9a3df519a7a256548e42b5bf08d5301fa1c0978348e03e89dd"} Oct 31 04:00:00.726889 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.726838 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" podStartSLOduration=1.726821878 podStartE2EDuration="1.726821878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:00.72449997 +0000 UTC m=+2624.919023093" watchObservedRunningTime="2026-10-31 04:00:00.726821878 +0000 UTC m=+2624.921344999" Oct 31 04:00:00.757267 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.755605 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64"] Oct 31 04:00:00.758254 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:00:00.758171 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1aacc1_81da_4014_8405_aaabadd1a281.slice/crio-de109d8a9ab023616218b22461bbc798e557b76277fca2eb5fa903381404d3a1 WatchSource:0}: Error finding container de109d8a9ab023616218b22461bbc798e557b76277fca2eb5fa903381404d3a1: Status 404 returned error can't find the container with id de109d8a9ab023616218b22461bbc798e557b76277fca2eb5fa903381404d3a1 Oct 31 04:00:00.928618 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.928582 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g"] Oct 31 04:00:00.932719 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:00.932681 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7f3159-rcf6g"] Oct 31 04:00:01.715061 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:01.715026 5520 generic.go:356] "Generic (PLEG): container finished" podID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerID="8c1a63b0aa99c1bd4049f07174844f18f2d485e14c81275f514a0570275cd8df" exitCode=60 Oct 31 04:00:01.715668 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:01.715115 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" event={"ID":"bc1aacc1-81da-4014-8405-aaabadd1a281","Type":"ContainerDied","Data":"8c1a63b0aa99c1bd4049f07174844f18f2d485e14c81275f514a0570275cd8df"} Oct 31 04:00:01.715668 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:01.715172 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" event={"ID":"bc1aacc1-81da-4014-8405-aaabadd1a281","Type":"ContainerStarted","Data":"de109d8a9ab023616218b22461bbc798e557b76277fca2eb5fa903381404d3a1"} Oct 31 04:00:01.715668 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:01.715456 5520 scope.go:122] "RemoveContainer" containerID="8c1a63b0aa99c1bd4049f07174844f18f2d485e14c81275f514a0570275cd8df" Oct 31 04:00:01.717320 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:01.717301 5520 generic.go:356] "Generic (PLEG): container finished" podID="a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" containerID="051a8f2c0b4afadadddd0804c34056893852fa62a69d737918fc8ff5387f8761" exitCode=0 Oct 31 04:00:01.717429 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:01.717364 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" event={"ID":"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8","Type":"ContainerDied","Data":"051a8f2c0b4afadadddd0804c34056893852fa62a69d737918fc8ff5387f8761"} Oct 31 04:00:01.917236 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:01.917179 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328a4da7-1ae6-4172-a251-fc30e4b16da1" path="/var/lib/kubelet/pods/328a4da7-1ae6-4172-a251-fc30e4b16da1/volumes" Oct 31 04:00:02.724864 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.724832 5520 generic.go:356] "Generic (PLEG): container finished" podID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerID="ffad84a5ca6dc5a148d96dd7b87f99d92e38e0d1de8a7bfddbd3b7bbf40569d1" exitCode=60 Oct 31 04:00:02.725346 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.724919 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" event={"ID":"bc1aacc1-81da-4014-8405-aaabadd1a281","Type":"ContainerDied","Data":"ffad84a5ca6dc5a148d96dd7b87f99d92e38e0d1de8a7bfddbd3b7bbf40569d1"} Oct 31 04:00:02.725346 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.724978 5520 scope.go:122] "RemoveContainer" containerID="8c1a63b0aa99c1bd4049f07174844f18f2d485e14c81275f514a0570275cd8df" Oct 31 04:00:02.725346 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.725257 5520 scope.go:122] "RemoveContainer" containerID="ffad84a5ca6dc5a148d96dd7b87f99d92e38e0d1de8a7bfddbd3b7bbf40569d1" Oct 31 04:00:02.725897 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:02.725501 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-cwh64_odh-ai-gateway-infra(bc1aacc1-81da-4014-8405-aaabadd1a281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" Oct 31 04:00:02.863689 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.863663 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 04:00:02.993524 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.993471 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-kube-api-access-jsgmc\" (UniqueName: \"kubernetes.io/projected/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-kube-api-access-jsgmc\") pod \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " Oct 31 04:00:02.993741 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.993628 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-xks-ca\") pod \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " Oct 31 04:00:02.993741 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.993689 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-openshift-ca\") pod \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\" (UID: \"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8\") " Oct 31 04:00:02.994075 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.994047 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-openshift-ca" pod "a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" (UID: "a8f8b9be-b4aa-490a-8c32-5c89c913f8c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:02.995882 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.995847 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-kube-api-access-jsgmc" pod "a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" (UID: "a8f8b9be-b4aa-490a-8c32-5c89c913f8c8"). InnerVolumeSpecName "kube-api-access-jsgmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:02.995882 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:02.995865 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-xks-ca" pod "a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" (UID: "a8f8b9be-b4aa-490a-8c32-5c89c913f8c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:03.095124 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:03.095014 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgmc\" (UniqueName: \"kubernetes.io/projected/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-kube-api-access-jsgmc\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:03.095124 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:03.095052 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:03.095124 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:03.095066 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:03.736438 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:03.736405 5520 scope.go:122] "RemoveContainer" containerID="ffad84a5ca6dc5a148d96dd7b87f99d92e38e0d1de8a7bfddbd3b7bbf40569d1" Oct 31 04:00:03.736996 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:03.736682 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-cwh64_odh-ai-gateway-infra(bc1aacc1-81da-4014-8405-aaabadd1a281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" Oct 31 04:00:03.738832 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:03.738806 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" Oct 31 04:00:03.738945 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:03.738814 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5" event={"ID":"a8f8b9be-b4aa-490a-8c32-5c89c913f8c8","Type":"ContainerDied","Data":"83ede8833f1dbc9a3df519a7a256548e42b5bf08d5301fa1c0978348e03e89dd"} Oct 31 04:00:03.738945 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:03.738929 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ede8833f1dbc9a3df519a7a256548e42b5bf08d5301fa1c0978348e03e89dd" Oct 31 04:00:04.431272 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:04.431235 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj"] Oct 31 04:00:05.071926 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:05.071884 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" podUID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:00:08.837528 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:08.837468 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf"] Oct 31 04:00:08.838900 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:08.838654 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" containerID="cri-o://758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c" gracePeriod=30 Oct 31 04:00:09.357109 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:09.357073 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:00:09.806478 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:09.806443 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5"] Oct 31 04:00:10.528801 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.528759 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx"] Oct 31 04:00:10.529382 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.529363 5520 cpu_manager.go:414] "containerMap: removing container" podUID="a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" containerName="revoke-keys" Oct 31 04:00:10.529454 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.529385 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" containerName="revoke-keys" Oct 31 04:00:10.529492 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.529470 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" containerName="revoke-keys" Oct 31 04:00:10.532527 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.532512 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.540472 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.540426 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:00:10.541041 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.540980 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx"] Oct 31 04:00:10.567928 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.567896 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.568110 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.567947 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.568110 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.568048 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpvq\" (UniqueName: \"kubernetes.io/projected/c8b1acaf-1867-4e02-a7fa-e40944707568-kube-api-access-wjpvq\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.669123 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.669083 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.669373 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.669141 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.669373 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.669175 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpvq\" (UniqueName: \"kubernetes.io/projected/c8b1acaf-1867-4e02-a7fa-e40944707568-kube-api-access-wjpvq\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.669858 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.669831 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.671784 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.671756 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.677742 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.677717 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpvq\" (UniqueName: \"kubernetes.io/projected/c8b1acaf-1867-4e02-a7fa-e40944707568-kube-api-access-wjpvq\") pod \"maas-api-revoke-sub-232b27ae-f76lx\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:10.759089 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.759045 5520 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:00:10.759284 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.759119 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:00:10.848048 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:10.847966 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:11.121711 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:11.121683 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx"] Oct 31 04:00:11.124559 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:00:11.124522 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b1acaf_1867_4e02_a7fa_e40944707568.slice/crio-0f99c776e9f1e230862a54e320f41318c721492c3a3f165b95ef971f1759e3eb WatchSource:0}: Error finding container 0f99c776e9f1e230862a54e320f41318c721492c3a3f165b95ef971f1759e3eb: Status 404 returned error can't find the container with id 0f99c776e9f1e230862a54e320f41318c721492c3a3f165b95ef971f1759e3eb Oct 31 04:00:11.787352 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:11.787315 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" event={"ID":"c8b1acaf-1867-4e02-a7fa-e40944707568","Type":"ContainerStarted","Data":"280e89d0008a738c2f906cece273c5ac4cecdcae7a7718d50f592f074a175b5c"} Oct 31 04:00:11.787780 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:11.787372 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" event={"ID":"c8b1acaf-1867-4e02-a7fa-e40944707568","Type":"ContainerStarted","Data":"0f99c776e9f1e230862a54e320f41318c721492c3a3f165b95ef971f1759e3eb"} Oct 31 04:00:11.806511 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:11.806459 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" podStartSLOduration=1.806444082 podStartE2EDuration="1.806444082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:11.803721254 +0000 UTC m=+2635.998244377" watchObservedRunningTime="2026-10-31 04:00:11.806444082 +0000 UTC m=+2636.000967207" Oct 31 04:00:12.795701 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:12.795665 5520 generic.go:356] "Generic (PLEG): container finished" podID="c8b1acaf-1867-4e02-a7fa-e40944707568" containerID="280e89d0008a738c2f906cece273c5ac4cecdcae7a7718d50f592f074a175b5c" exitCode=0 Oct 31 04:00:12.796229 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:12.795754 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" event={"ID":"c8b1acaf-1867-4e02-a7fa-e40944707568","Type":"ContainerDied","Data":"280e89d0008a738c2f906cece273c5ac4cecdcae7a7718d50f592f074a175b5c"} Oct 31 04:00:13.964878 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:13.964849 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:14.005762 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.005730 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-xks-ca\") pod \"c8b1acaf-1867-4e02-a7fa-e40944707568\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " Oct 31 04:00:14.005978 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.005837 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b1acaf-1867-4e02-a7fa-e40944707568-kube-api-access-wjpvq\" (UniqueName: \"kubernetes.io/projected/c8b1acaf-1867-4e02-a7fa-e40944707568-kube-api-access-wjpvq\") pod \"c8b1acaf-1867-4e02-a7fa-e40944707568\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " Oct 31 04:00:14.005978 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.005934 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-openshift-ca\") pod \"c8b1acaf-1867-4e02-a7fa-e40944707568\" (UID: \"c8b1acaf-1867-4e02-a7fa-e40944707568\") " Oct 31 04:00:14.006427 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.006380 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-openshift-ca" pod "c8b1acaf-1867-4e02-a7fa-e40944707568" (UID: "c8b1acaf-1867-4e02-a7fa-e40944707568"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:14.008249 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.008189 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-xks-ca" pod "c8b1acaf-1867-4e02-a7fa-e40944707568" (UID: "c8b1acaf-1867-4e02-a7fa-e40944707568"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:14.008541 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.008521 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b1acaf-1867-4e02-a7fa-e40944707568-kube-api-access-wjpvq" pod "c8b1acaf-1867-4e02-a7fa-e40944707568" (UID: "c8b1acaf-1867-4e02-a7fa-e40944707568"). InnerVolumeSpecName "kube-api-access-wjpvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:14.094827 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.094800 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 04:00:14.106948 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.106924 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.106948 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.106951 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1acaf-1867-4e02-a7fa-e40944707568-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.107120 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.106961 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjpvq\" (UniqueName: \"kubernetes.io/projected/c8b1acaf-1867-4e02-a7fa-e40944707568-kube-api-access-wjpvq\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.207936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.207902 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-credential-socket\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.207936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.207948 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-data\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208244 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208050 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d435696-c420-4d28-84bb-550440e03636-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d435696-c420-4d28-84bb-550440e03636-istiod-ca-cert\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208244 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208087 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-certs\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208244 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208139 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-envoy\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208244 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208196 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-kube-api-access-sqkqg\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-kube-api-access-sqkqg\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208470 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208269 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d435696-c420-4d28-84bb-550440e03636-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d435696-c420-4d28-84bb-550440e03636-istio-podinfo\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208470 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208307 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-socket\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208470 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208319 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-data" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:14.208470 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208350 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-istio-token\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-istio-token\") pod \"3d435696-c420-4d28-84bb-550440e03636\" (UID: \"3d435696-c420-4d28-84bb-550440e03636\") " Oct 31 04:00:14.208703 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208563 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d435696-c420-4d28-84bb-550440e03636-istiod-ca-cert" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:14.208703 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208643 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-credential-socket" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:14.208824 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208718 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d435696-c420-4d28-84bb-550440e03636-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.208824 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208737 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.208996 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.208965 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-socket" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:14.209134 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.209023 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-certs" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:14.210976 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.210950 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-kube-api-access-sqkqg" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "kube-api-access-sqkqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:14.211091 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.211019 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d435696-c420-4d28-84bb-550440e03636-istio-podinfo" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:14.211187 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.211167 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-envoy" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:14.211312 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.211186 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-istio-token" pod "3d435696-c420-4d28-84bb-550440e03636" (UID: "3d435696-c420-4d28-84bb-550440e03636"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:14.309438 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.309396 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.309438 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.309435 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.309673 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.309450 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqkqg\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-kube-api-access-sqkqg\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.309673 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.309466 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d435696-c420-4d28-84bb-550440e03636-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.309673 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.309482 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.309673 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.309495 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d435696-c420-4d28-84bb-550440e03636-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.309673 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.309509 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d435696-c420-4d28-84bb-550440e03636-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:14.807326 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.807258 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" podUID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" containerName="payload-processing" containerID="cri-o://ebd41b63858b5c2597d18c87fc8d100ac469dd53e3164947bc53960cd0576f32" gracePeriod=25 Oct 31 04:00:14.812390 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.812356 5520 generic.go:356] "Generic (PLEG): container finished" podID="3d435696-c420-4d28-84bb-550440e03636" containerID="758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c" exitCode=0 Oct 31 04:00:14.812566 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.812542 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" event={"ID":"3d435696-c420-4d28-84bb-550440e03636","Type":"ContainerDied","Data":"758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c"} Oct 31 04:00:14.812649 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.812604 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" event={"ID":"3d435696-c420-4d28-84bb-550440e03636","Type":"ContainerDied","Data":"3dab127d2bb97a40c28c987e3e803ddec6a169148c6464ed1c1d0d4974d6ef53"} Oct 31 04:00:14.812649 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.812632 5520 scope.go:122] "RemoveContainer" containerID="758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c" Oct 31 04:00:14.812805 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.812792 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf" Oct 31 04:00:14.825314 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.825286 5520 generic.go:356] "Generic (PLEG): container finished" podID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" containerID="b93e9b6e343de1f1b1d0a7e3de512664410f574b65880bd768a4596cc4a8d7fa" exitCode=0 Oct 31 04:00:14.825539 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.825361 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" event={"ID":"bc45b5c9-db98-4f39-93ca-18fdf3b15db9","Type":"ContainerDied","Data":"b93e9b6e343de1f1b1d0a7e3de512664410f574b65880bd768a4596cc4a8d7fa"} Oct 31 04:00:14.827086 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.827053 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" event={"ID":"c8b1acaf-1867-4e02-a7fa-e40944707568","Type":"ContainerDied","Data":"0f99c776e9f1e230862a54e320f41318c721492c3a3f165b95ef971f1759e3eb"} Oct 31 04:00:14.827299 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.827107 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx" Oct 31 04:00:14.827299 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.827108 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f99c776e9f1e230862a54e320f41318c721492c3a3f165b95ef971f1759e3eb" Oct 31 04:00:14.836351 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.836328 5520 scope.go:122] "RemoveContainer" containerID="5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22" Oct 31 04:00:14.842820 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.842786 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf"] Oct 31 04:00:14.847824 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.847801 5520 scope.go:122] "RemoveContainer" containerID="758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c" Oct 31 04:00:14.848123 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.848090 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-t6rnf"] Oct 31 04:00:14.848267 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:14.848244 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c\": container with ID starting with 758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c not found: ID does not exist" containerID="758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c" Oct 31 04:00:14.848355 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.848281 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c"} err="failed to get container status \"758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c\": rpc error: code = NotFound desc = could not find container \"758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c\": container with ID starting with 758a251f1224a966a5df181f1279615106e4fe70bedf33d6a6e3be258bdcf99c not found: ID does not exist" Oct 31 04:00:14.848355 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.848308 5520 scope.go:122] "RemoveContainer" containerID="5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22" Oct 31 04:00:14.848669 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:14.848639 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22\": container with ID starting with 5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22 not found: ID does not exist" containerID="5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22" Oct 31 04:00:14.848764 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.848675 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22"} err="failed to get container status \"5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22\": rpc error: code = NotFound desc = could not find container \"5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22\": container with ID starting with 5e8942fccdad37fb15b9c6d6a24ad7fb1922fb5d6b3349adf8b3861b02267e22 not found: ID does not exist" Oct 31 04:00:14.860903 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.860883 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 04:00:14.923553 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.923519 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-plugins-config-volume\") pod \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " Oct 31 04:00:14.923761 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.923593 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-kube-api-access-sfhx4\" (UniqueName: \"kubernetes.io/projected/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-kube-api-access-sfhx4\") pod \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\" (UID: \"bc45b5c9-db98-4f39-93ca-18fdf3b15db9\") " Oct 31 04:00:14.923902 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.923875 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-plugins-config-volume" pod "bc45b5c9-db98-4f39-93ca-18fdf3b15db9" (UID: "bc45b5c9-db98-4f39-93ca-18fdf3b15db9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:14.926107 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:14.926084 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-kube-api-access-sfhx4" pod "bc45b5c9-db98-4f39-93ca-18fdf3b15db9" (UID: "bc45b5c9-db98-4f39-93ca-18fdf3b15db9"). InnerVolumeSpecName "kube-api-access-sfhx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:15.024550 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.024513 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfhx4\" (UniqueName: \"kubernetes.io/projected/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-kube-api-access-sfhx4\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:15.024550 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.024546 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc45b5c9-db98-4f39-93ca-18fdf3b15db9-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:15.839037 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.839000 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" Oct 31 04:00:15.839279 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.839038 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9" event={"ID":"bc45b5c9-db98-4f39-93ca-18fdf3b15db9","Type":"ContainerDied","Data":"a011a7cf4b7b7f1db396de9eca838964ff541c6f0f1acf4692f7df61974ca92d"} Oct 31 04:00:15.839279 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.839124 5520 scope.go:122] "RemoveContainer" containerID="b93e9b6e343de1f1b1d0a7e3de512664410f574b65880bd768a4596cc4a8d7fa" Oct 31 04:00:15.861474 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.861436 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9"] Oct 31 04:00:15.863895 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.863869 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-ab21e4-59c54fd55-ql2n9"] Oct 31 04:00:15.914634 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.914599 5520 scope.go:122] "RemoveContainer" containerID="ffad84a5ca6dc5a148d96dd7b87f99d92e38e0d1de8a7bfddbd3b7bbf40569d1" Oct 31 04:00:15.921956 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.921185 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d435696-c420-4d28-84bb-550440e03636" path="/var/lib/kubelet/pods/3d435696-c420-4d28-84bb-550440e03636/volumes" Oct 31 04:00:15.921956 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:15.921790 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" path="/var/lib/kubelet/pods/bc45b5c9-db98-4f39-93ca-18fdf3b15db9/volumes" Oct 31 04:00:16.847349 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:16.847318 5520 generic.go:356] "Generic (PLEG): container finished" podID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerID="8c9e0d4cb44072e8da077cbb92e6ba2faaae04ce9353227fa0fd78bc048c3392" exitCode=60 Oct 31 04:00:16.847846 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:16.847394 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" event={"ID":"bc1aacc1-81da-4014-8405-aaabadd1a281","Type":"ContainerDied","Data":"8c9e0d4cb44072e8da077cbb92e6ba2faaae04ce9353227fa0fd78bc048c3392"} Oct 31 04:00:16.847846 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:16.847468 5520 scope.go:122] "RemoveContainer" containerID="ffad84a5ca6dc5a148d96dd7b87f99d92e38e0d1de8a7bfddbd3b7bbf40569d1" Oct 31 04:00:16.847975 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:16.847844 5520 scope.go:122] "RemoveContainer" containerID="8c9e0d4cb44072e8da077cbb92e6ba2faaae04ce9353227fa0fd78bc048c3392" Oct 31 04:00:16.848104 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:16.848083 5520 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-cwh64_odh-ai-gateway-infra(bc1aacc1-81da-4014-8405-aaabadd1a281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" Oct 31 04:00:17.872542 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:17.872506 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64"] Oct 31 04:00:18.015432 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.015406 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:18.157328 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.157243 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1aacc1-81da-4014-8405-aaabadd1a281-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1aacc1-81da-4014-8405-aaabadd1a281-maas-api-service-ca\") pod \"bc1aacc1-81da-4014-8405-aaabadd1a281\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " Oct 31 04:00:18.157492 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.157386 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1aacc1-81da-4014-8405-aaabadd1a281-kube-api-access-9r7zl\" (UniqueName: \"kubernetes.io/projected/bc1aacc1-81da-4014-8405-aaabadd1a281-kube-api-access-9r7zl\") pod \"bc1aacc1-81da-4014-8405-aaabadd1a281\" (UID: \"bc1aacc1-81da-4014-8405-aaabadd1a281\") " Oct 31 04:00:18.157625 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.157599 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1aacc1-81da-4014-8405-aaabadd1a281-maas-api-service-ca" pod "bc1aacc1-81da-4014-8405-aaabadd1a281" (UID: "bc1aacc1-81da-4014-8405-aaabadd1a281"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:18.159609 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.159582 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1aacc1-81da-4014-8405-aaabadd1a281-kube-api-access-9r7zl" pod "bc1aacc1-81da-4014-8405-aaabadd1a281" (UID: "bc1aacc1-81da-4014-8405-aaabadd1a281"). InnerVolumeSpecName "kube-api-access-9r7zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:18.258938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.258904 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r7zl\" (UniqueName: \"kubernetes.io/projected/bc1aacc1-81da-4014-8405-aaabadd1a281-kube-api-access-9r7zl\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:18.258938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.258932 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1aacc1-81da-4014-8405-aaabadd1a281-maas-api-service-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:18.861059 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.861028 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" Oct 31 04:00:18.861059 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.861053 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64" event={"ID":"bc1aacc1-81da-4014-8405-aaabadd1a281","Type":"ContainerDied","Data":"de109d8a9ab023616218b22461bbc798e557b76277fca2eb5fa903381404d3a1"} Oct 31 04:00:18.861349 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.861120 5520 scope.go:122] "RemoveContainer" containerID="8c9e0d4cb44072e8da077cbb92e6ba2faaae04ce9353227fa0fd78bc048c3392" Oct 31 04:00:18.884733 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.884704 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64"] Oct 31 04:00:18.892599 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:18.892577 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-cwh64"] Oct 31 04:00:19.457131 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.457064 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" containerID="cri-o://a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982" gracePeriod=45 Oct 31 04:00:19.705898 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.705867 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 04:00:19.774450 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774363 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tmp-dir\") pod \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " Oct 31 04:00:19.774450 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774410 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tls-certs\") pod \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " Oct 31 04:00:19.774700 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774490 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-kube-api-access-pzn2b\" (UniqueName: \"kubernetes.io/projected/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-kube-api-access-pzn2b\") pod \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " Oct 31 04:00:19.774700 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774521 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-dshm\") pod \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " Oct 31 04:00:19.774700 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774593 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-home\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-home\") pod \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " Oct 31 04:00:19.774700 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774648 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-model-cache\") pod \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\" (UID: \"1f6ddb2f-0908-47ff-8eca-76242df4ae8d\") " Oct 31 04:00:19.774700 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774677 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tmp-dir" pod "1f6ddb2f-0908-47ff-8eca-76242df4ae8d" (UID: "1f6ddb2f-0908-47ff-8eca-76242df4ae8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.775055 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.774919 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.775194 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.775172 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-model-cache" pod "1f6ddb2f-0908-47ff-8eca-76242df4ae8d" (UID: "1f6ddb2f-0908-47ff-8eca-76242df4ae8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.775632 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.775580 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-home" pod "1f6ddb2f-0908-47ff-8eca-76242df4ae8d" (UID: "1f6ddb2f-0908-47ff-8eca-76242df4ae8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.777761 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.777733 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-kube-api-access-pzn2b" pod "1f6ddb2f-0908-47ff-8eca-76242df4ae8d" (UID: "1f6ddb2f-0908-47ff-8eca-76242df4ae8d"). InnerVolumeSpecName "kube-api-access-pzn2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:19.778053 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.778033 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tls-certs" pod "1f6ddb2f-0908-47ff-8eca-76242df4ae8d" (UID: "1f6ddb2f-0908-47ff-8eca-76242df4ae8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:19.778053 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.778043 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-dshm" pod "1f6ddb2f-0908-47ff-8eca-76242df4ae8d" (UID: "1f6ddb2f-0908-47ff-8eca-76242df4ae8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.868369 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.868335 5520 generic.go:356] "Generic (PLEG): container finished" podID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerID="a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982" exitCode=0 Oct 31 04:00:19.868541 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.868383 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" Oct 31 04:00:19.868541 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.868413 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" event={"ID":"1f6ddb2f-0908-47ff-8eca-76242df4ae8d","Type":"ContainerDied","Data":"a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982"} Oct 31 04:00:19.868541 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.868456 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj" event={"ID":"1f6ddb2f-0908-47ff-8eca-76242df4ae8d","Type":"ContainerDied","Data":"9bbf16162e57487a66618257f6d456e2ff6402c524646dea82902c0594a65aaf"} Oct 31 04:00:19.868541 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.868475 5520 scope.go:122] "RemoveContainer" containerID="a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982" Oct 31 04:00:19.875420 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.875399 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.875420 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.875422 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzn2b\" (UniqueName: \"kubernetes.io/projected/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-kube-api-access-pzn2b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.875594 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.875432 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.875594 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.875441 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.875594 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.875448 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6ddb2f-0908-47ff-8eca-76242df4ae8d-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.878136 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.878115 5520 scope.go:122] "RemoveContainer" containerID="a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982" Oct 31 04:00:19.878403 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:19.878386 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982\": container with ID starting with a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982 not found: ID does not exist" containerID="a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982" Oct 31 04:00:19.878473 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.878412 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982"} err="failed to get container status \"a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982\": rpc error: code = NotFound desc = could not find container \"a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982\": container with ID starting with a102fc119a2f84e10a4a7ae26ef53b1153e89f48ab17e98db24d434a303fa982 not found: ID does not exist" Oct 31 04:00:19.891773 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.891746 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj"] Oct 31 04:00:19.898415 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.898393 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/simulator-w5-bcc7a4-kserve-7db9449744-vgddj"] Oct 31 04:00:19.915168 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.915141 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" path="/var/lib/kubelet/pods/1f6ddb2f-0908-47ff-8eca-76242df4ae8d/volumes" Oct 31 04:00:19.915548 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:19.915534 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" path="/var/lib/kubelet/pods/bc1aacc1-81da-4014-8405-aaabadd1a281/volumes" Oct 31 04:00:23.383854 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:23.383812 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" podUID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:00:25.824160 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:25.824117 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5"] Oct 31 04:00:25.826149 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:25.826124 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-bcc7a4-gskx5"] Oct 31 04:00:25.925758 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:25.925708 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f8b9be-b4aa-490a-8c32-5c89c913f8c8" path="/var/lib/kubelet/pods/a8f8b9be-b4aa-490a-8c32-5c89c913f8c8/volumes" Oct 31 04:00:32.381856 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:32.381817 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx"] Oct 31 04:00:32.382427 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:32.382071 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" containerID="cri-o://5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75" gracePeriod=30 Oct 31 04:00:33.379509 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:33.379464 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" podUID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:00:34.957758 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:34.957724 5520 generic.go:356] "Generic (PLEG): container finished" podID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" containerID="ebd41b63858b5c2597d18c87fc8d100ac469dd53e3164947bc53960cd0576f32" exitCode=0 Oct 31 04:00:34.958179 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:34.957799 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" event={"ID":"57ba03ee-18ea-42dc-a954-8d6b3ab652cd","Type":"ContainerDied","Data":"ebd41b63858b5c2597d18c87fc8d100ac469dd53e3164947bc53960cd0576f32"} Oct 31 04:00:35.061382 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.061354 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 04:00:35.145967 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.145881 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-plugins-config-volume\") pod \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " Oct 31 04:00:35.146127 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.145966 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-kube-api-access-mntdv\" (UniqueName: \"kubernetes.io/projected/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-kube-api-access-mntdv\") pod \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\" (UID: \"57ba03ee-18ea-42dc-a954-8d6b3ab652cd\") " Oct 31 04:00:35.146346 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.146326 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-plugins-config-volume" pod "57ba03ee-18ea-42dc-a954-8d6b3ab652cd" (UID: "57ba03ee-18ea-42dc-a954-8d6b3ab652cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:35.148274 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.148246 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-kube-api-access-mntdv" pod "57ba03ee-18ea-42dc-a954-8d6b3ab652cd" (UID: "57ba03ee-18ea-42dc-a954-8d6b3ab652cd"). InnerVolumeSpecName "kube-api-access-mntdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:35.247185 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.247145 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:35.247185 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.247176 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mntdv\" (UniqueName: \"kubernetes.io/projected/57ba03ee-18ea-42dc-a954-8d6b3ab652cd-kube-api-access-mntdv\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:35.963724 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.963695 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" Oct 31 04:00:35.964267 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.963687 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5" event={"ID":"57ba03ee-18ea-42dc-a954-8d6b3ab652cd","Type":"ContainerDied","Data":"286c350a89a4aa997da37c2dcebeaf64c3473215599af31bd23df3a2eec466e5"} Oct 31 04:00:35.964267 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.963837 5520 scope.go:122] "RemoveContainer" containerID="ebd41b63858b5c2597d18c87fc8d100ac469dd53e3164947bc53960cd0576f32" Oct 31 04:00:35.984882 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.984849 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5"] Oct 31 04:00:35.985004 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:35.984905 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-bcc7a4-749cfb845c-smgz5"] Oct 31 04:00:37.751038 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.751005 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.767951 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-socket\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768021 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-certs\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768053 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-credential-socket\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768081 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-data\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768138 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-token\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-token\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768172 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1306dc-5094-4f7d-ad20-667064dd914c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1306dc-5094-4f7d-ad20-667064dd914c-istiod-ca-cert\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768252 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-kube-api-access-f7hgz\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-kube-api-access-f7hgz\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768283 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-podinfo\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.769057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.768312 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-envoy\") pod \"5c1306dc-5094-4f7d-ad20-667064dd914c\" (UID: \"5c1306dc-5094-4f7d-ad20-667064dd914c\") " Oct 31 04:00:37.776007 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.773482 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-socket" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:37.776007 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.773817 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-certs" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:37.776007 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.775969 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-credential-socket" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:37.776537 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.776497 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1306dc-5094-4f7d-ad20-667064dd914c-istiod-ca-cert" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:37.776925 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.776906 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-data" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:37.797815 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.797759 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-kube-api-access-f7hgz" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "kube-api-access-f7hgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:37.798040 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.798002 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-token" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:37.798158 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.798137 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-podinfo" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:37.819684 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.819619 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-envoy" pod "5c1306dc-5094-4f7d-ad20-667064dd914c" (UID: "5c1306dc-5094-4f7d-ad20-667064dd914c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:37.869569 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869523 5520 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-data\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869569 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869568 5520 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-token\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869569 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869584 5520 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1306dc-5094-4f7d-ad20-667064dd914c-istiod-ca-cert\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869601 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7hgz\" (UniqueName: \"kubernetes.io/projected/5c1306dc-5094-4f7d-ad20-667064dd914c-kube-api-access-f7hgz\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869617 5520 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-podinfo\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869630 5520 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-istio-envoy\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869645 5520 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869658 5520 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-workload-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.869938 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.869672 5520 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1306dc-5094-4f7d-ad20-667064dd914c-credential-socket\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:37.917933 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.917881 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" path="/var/lib/kubelet/pods/57ba03ee-18ea-42dc-a954-8d6b3ab652cd/volumes" Oct 31 04:00:37.981122 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.981031 5520 generic.go:356] "Generic (PLEG): container finished" podID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerID="5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75" exitCode=0 Oct 31 04:00:37.981412 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.981100 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" Oct 31 04:00:37.981606 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.981123 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" event={"ID":"5c1306dc-5094-4f7d-ad20-667064dd914c","Type":"ContainerDied","Data":"5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75"} Oct 31 04:00:37.981794 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.981739 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx" event={"ID":"5c1306dc-5094-4f7d-ad20-667064dd914c","Type":"ContainerDied","Data":"e43fcf33ff648c8eb111d31ed98ab80ddd60cfee74ed8888a43ca3c899b6c449"} Oct 31 04:00:37.981996 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:37.981979 5520 scope.go:122] "RemoveContainer" containerID="5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75" Oct 31 04:00:38.004959 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:38.004565 5520 scope.go:122] "RemoveContainer" containerID="5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75" Oct 31 04:00:38.005492 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:38.005386 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75\": container with ID starting with 5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75 not found: ID does not exist" containerID="5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75" Oct 31 04:00:38.005492 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:38.005431 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75"} err="failed to get container status \"5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75\": rpc error: code = NotFound desc = could not find container \"5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75\": container with ID starting with 5d0013a39a558a8b446cb42a0d0d55e484157864c54301039096f0a2a8785e75 not found: ID does not exist" Oct 31 04:00:38.006034 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:38.005973 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx"] Oct 31 04:00:38.008707 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:38.008682 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ld6jx"] Oct 31 04:00:39.357038 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:00:39.356995 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:00:39.914735 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:39.914704 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" path="/var/lib/kubelet/pods/5c1306dc-5094-4f7d-ad20-667064dd914c/volumes" Oct 31 04:00:55.306663 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.306619 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs"] Oct 31 04:00:55.307171 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307154 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" Oct 31 04:00:55.307240 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307173 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" Oct 31 04:00:55.307240 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307191 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" containerName="payload-pre-processing" Oct 31 04:00:55.307240 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307216 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" containerName="payload-pre-processing" Oct 31 04:00:55.307240 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307233 5520 cpu_manager.go:414] "containerMap: removing container" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" Oct 31 04:00:55.307240 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307239 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307254 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307261 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307275 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307281 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307296 5520 cpu_manager.go:414] "containerMap: removing container" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307303 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307311 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307318 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307328 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307335 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307352 5520 cpu_manager.go:414] "containerMap: removing container" podUID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" containerName="payload-processing" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307359 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" containerName="payload-processing" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307369 5520 cpu_manager.go:414] "containerMap: removing container" podUID="c8b1acaf-1867-4e02-a7fa-e40944707568" containerName="revoke-keys" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307375 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b1acaf-1867-4e02-a7fa-e40944707568" containerName="revoke-keys" Oct 31 04:00:55.307460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307458 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307471 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc45b5c9-db98-4f39-93ca-18fdf3b15db9" containerName="payload-pre-processing" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307482 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d435696-c420-4d28-84bb-550440e03636" containerName="istio-proxy" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307491 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307500 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307508 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6ddb2f-0908-47ff-8eca-76242df4ae8d" containerName="main" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307516 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1aacc1-81da-4014-8405-aaabadd1a281" containerName="cleanup" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307525 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1306dc-5094-4f7d-ad20-667064dd914c" containerName="istio-proxy" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307533 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ba03ee-18ea-42dc-a954-8d6b3ab652cd" containerName="payload-processing" Oct 31 04:00:55.307910 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.307541 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b1acaf-1867-4e02-a7fa-e40944707568" containerName="revoke-keys" Oct 31 04:00:55.310754 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.310736 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.318658 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.318623 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:00:55.319135 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.318918 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:00:55.319135 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.319033 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:00:55.319643 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.319624 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:00:55.319790 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.319665 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs"] Oct 31 04:00:55.433871 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.433827 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.434090 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.433985 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2c7z\" (UniqueName: \"kubernetes.io/projected/14f16c06-8bcb-43e9-aef5-0bcde3c18356-kube-api-access-m2c7z\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.434090 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.434055 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.535287 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.535249 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.535472 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.535336 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2c7z\" (UniqueName: \"kubernetes.io/projected/14f16c06-8bcb-43e9-aef5-0bcde3c18356-kube-api-access-m2c7z\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.535472 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.535369 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.536118 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.536034 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.537856 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.537834 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.544320 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.544296 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2c7z\" (UniqueName: \"kubernetes.io/projected/14f16c06-8bcb-43e9-aef5-0bcde3c18356-kube-api-access-m2c7z\") pod \"maas-api-revoke-sub-85237e45-p74cs\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.625673 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.625595 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:55.900867 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:55.900844 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs"] Oct 31 04:00:55.903422 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:00:55.903391 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f16c06_8bcb_43e9_aef5_0bcde3c18356.slice/crio-6108458c43dc78da20dcf4384ad49d6265b680d147f174091bcbbdfabe4a1c6b WatchSource:0}: Error finding container 6108458c43dc78da20dcf4384ad49d6265b680d147f174091bcbbdfabe4a1c6b: Status 404 returned error can't find the container with id 6108458c43dc78da20dcf4384ad49d6265b680d147f174091bcbbdfabe4a1c6b Oct 31 04:00:56.074813 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:56.074767 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" event={"ID":"14f16c06-8bcb-43e9-aef5-0bcde3c18356","Type":"ContainerStarted","Data":"6108458c43dc78da20dcf4384ad49d6265b680d147f174091bcbbdfabe4a1c6b"} Oct 31 04:00:57.081661 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:57.081554 5520 generic.go:356] "Generic (PLEG): container finished" podID="14f16c06-8bcb-43e9-aef5-0bcde3c18356" containerID="d7d9fcded7a4412afa03f9f71a28480763d102ee70e1d074f0fc80601a33a27e" exitCode=0 Oct 31 04:00:57.082145 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:57.081654 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" event={"ID":"14f16c06-8bcb-43e9-aef5-0bcde3c18356","Type":"ContainerDied","Data":"d7d9fcded7a4412afa03f9f71a28480763d102ee70e1d074f0fc80601a33a27e"} Oct 31 04:00:58.216869 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.216843 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:58.361268 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.361153 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f16c06-8bcb-43e9-aef5-0bcde3c18356-kube-api-access-m2c7z\" (UniqueName: \"kubernetes.io/projected/14f16c06-8bcb-43e9-aef5-0bcde3c18356-kube-api-access-m2c7z\") pod \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " Oct 31 04:00:58.361436 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.361319 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-openshift-ca\") pod \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " Oct 31 04:00:58.361436 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.361342 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-xks-ca\") pod \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\" (UID: \"14f16c06-8bcb-43e9-aef5-0bcde3c18356\") " Oct 31 04:00:58.361678 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.361655 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-openshift-ca" pod "14f16c06-8bcb-43e9-aef5-0bcde3c18356" (UID: "14f16c06-8bcb-43e9-aef5-0bcde3c18356"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:58.363519 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.363494 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-xks-ca" pod "14f16c06-8bcb-43e9-aef5-0bcde3c18356" (UID: "14f16c06-8bcb-43e9-aef5-0bcde3c18356"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:58.363519 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.363512 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f16c06-8bcb-43e9-aef5-0bcde3c18356-kube-api-access-m2c7z" pod "14f16c06-8bcb-43e9-aef5-0bcde3c18356" (UID: "14f16c06-8bcb-43e9-aef5-0bcde3c18356"). InnerVolumeSpecName "kube-api-access-m2c7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:58.462320 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.462281 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:58.462320 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.462313 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f16c06-8bcb-43e9-aef5-0bcde3c18356-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:58.462320 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:58.462323 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2c7z\" (UniqueName: \"kubernetes.io/projected/14f16c06-8bcb-43e9-aef5-0bcde3c18356-kube-api-access-m2c7z\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:59.093288 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:59.093257 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" Oct 31 04:00:59.093471 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:59.093259 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs" event={"ID":"14f16c06-8bcb-43e9-aef5-0bcde3c18356","Type":"ContainerDied","Data":"6108458c43dc78da20dcf4384ad49d6265b680d147f174091bcbbdfabe4a1c6b"} Oct 31 04:00:59.093471 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:00:59.093355 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6108458c43dc78da20dcf4384ad49d6265b680d147f174091bcbbdfabe4a1c6b" Oct 31 04:01:00.974635 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:00.974600 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch"] Oct 31 04:01:00.975157 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:00.975136 5520 cpu_manager.go:414] "containerMap: removing container" podUID="14f16c06-8bcb-43e9-aef5-0bcde3c18356" containerName="revoke-keys" Oct 31 04:01:00.975284 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:00.975161 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f16c06-8bcb-43e9-aef5-0bcde3c18356" containerName="revoke-keys" Oct 31 04:01:00.975357 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:00.975290 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f16c06-8bcb-43e9-aef5-0bcde3c18356" containerName="revoke-keys" Oct 31 04:01:00.979236 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:00.979188 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:00.994761 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:00.994728 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch"] Oct 31 04:01:01.090370 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090326 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090552 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090376 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090552 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090462 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktms\" (UniqueName: \"kubernetes.io/projected/6532e670-a319-423c-8f6f-ccea217ec5a2-kube-api-access-bktms\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090698 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090568 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090698 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090620 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090698 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090656 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090855 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090732 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090855 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090775 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.090855 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.090815 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6532e670-a319-423c-8f6f-ccea217ec5a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.191794 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.191754 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192014 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.191806 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192014 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.191837 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192014 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.191880 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192014 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.191901 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192014 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.191921 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6532e670-a319-423c-8f6f-ccea217ec5a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192317 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192038 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192317 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192092 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192317 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192119 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktms\" (UniqueName: \"kubernetes.io/projected/6532e670-a319-423c-8f6f-ccea217ec5a2-kube-api-access-bktms\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192497 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192360 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192497 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192434 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192633 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192610 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192777 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192752 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.192820 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.192796 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6532e670-a319-423c-8f6f-ccea217ec5a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.194344 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.194323 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.194479 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.194463 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.200712 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.200685 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6532e670-a319-423c-8f6f-ccea217ec5a2-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.200837 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.200687 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktms\" (UniqueName: \"kubernetes.io/projected/6532e670-a319-423c-8f6f-ccea217ec5a2-kube-api-access-bktms\") pod \"maas-default-gateway-openshift-default-769df7846c-kzdch\" (UID: \"6532e670-a319-423c-8f6f-ccea217ec5a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.332149 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.332062 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:01.612900 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.612873 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch"] Oct 31 04:01:01.614571 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:01:01.614544 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6532e670_a319_423c_8f6f_ccea217ec5a2.slice/crio-85422db92f236dab29c9d282e56a46f56db3a21aba2afd59a95b553f49f45a09 WatchSource:0}: Error finding container 85422db92f236dab29c9d282e56a46f56db3a21aba2afd59a95b553f49f45a09: Status 404 returned error can't find the container with id 85422db92f236dab29c9d282e56a46f56db3a21aba2afd59a95b553f49f45a09 Oct 31 04:01:01.616826 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.616794 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 04:01:01.616920 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.616864 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 04:01:01.616920 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:01.616891 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 04:01:02.109760 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:02.109720 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" event={"ID":"6532e670-a319-423c-8f6f-ccea217ec5a2","Type":"ContainerStarted","Data":"6407f8bee21b751cca2243a3f00d714273d28d9252919750458c38827dceae56"} Oct 31 04:01:02.110196 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:02.109776 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" event={"ID":"6532e670-a319-423c-8f6f-ccea217ec5a2","Type":"ContainerStarted","Data":"85422db92f236dab29c9d282e56a46f56db3a21aba2afd59a95b553f49f45a09"} Oct 31 04:01:02.131165 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:02.131110 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podStartSLOduration=2.1310971260000002 podStartE2EDuration="2.131097126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:02.129108005 +0000 UTC m=+2686.323631130" watchObservedRunningTime="2026-10-31 04:01:02.131097126 +0000 UTC m=+2686.325620243" Oct 31 04:01:02.333214 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:02.333180 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:02.334476 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:02.334441 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 04:01:02.334597 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:02.334489 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 04:01:03.332954 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:03.332913 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 04:01:03.333532 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:03.332980 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 04:01:04.333420 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:04.333390 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 04:01:04.333916 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:04.333440 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 04:01:05.332768 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:05.332729 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 04:01:05.332970 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:05.332785 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 04:01:06.332843 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:06.332801 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 04:01:06.333383 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:06.332861 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 04:01:07.332507 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:07.332470 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 04:01:07.332692 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:07.332532 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 04:01:08.332985 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:08.332943 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 04:01:08.333417 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:08.333029 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 04:01:09.356984 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:01:09.356937 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:01:09.776949 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:09.776919 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:09.777281 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:09.777257 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:09.778368 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:09.778342 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" Oct 31 04:01:16.051538 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.051497 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb"] Oct 31 04:01:16.055934 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.055906 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.070355 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.070326 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb"] Oct 31 04:01:16.146777 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.146738 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21db5238-717b-480c-ab5a-c1d81658a00e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.146986 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.146803 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.146986 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.146904 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.146986 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.146925 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.147140 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.146998 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.147140 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.147021 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21db5238-717b-480c-ab5a-c1d81658a00e-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.147140 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.147046 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.147140 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.147068 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21db5238-717b-480c-ab5a-c1d81658a00e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.147331 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.147166 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs96g\" (UniqueName: \"kubernetes.io/projected/21db5238-717b-480c-ab5a-c1d81658a00e-kube-api-access-vs96g\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.247715 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.247671 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs96g\" (UniqueName: \"kubernetes.io/projected/21db5238-717b-480c-ab5a-c1d81658a00e-kube-api-access-vs96g\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.247953 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.247727 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21db5238-717b-480c-ab5a-c1d81658a00e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.247953 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.247769 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.247953 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.247797 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248150 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.247940 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248150 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248014 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248150 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248045 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21db5238-717b-480c-ab5a-c1d81658a00e-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248150 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248075 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248150 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248109 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21db5238-717b-480c-ab5a-c1d81658a00e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248462 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248253 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248462 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248280 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248639 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248564 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.248639 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248591 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.249017 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.248996 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21db5238-717b-480c-ab5a-c1d81658a00e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.250505 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.250485 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21db5238-717b-480c-ab5a-c1d81658a00e-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.250650 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.250634 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21db5238-717b-480c-ab5a-c1d81658a00e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.255589 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.255564 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs96g\" (UniqueName: \"kubernetes.io/projected/21db5238-717b-480c-ab5a-c1d81658a00e-kube-api-access-vs96g\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.255687 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.255572 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21db5238-717b-480c-ab5a-c1d81658a00e-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-bdwhb\" (UID: \"21db5238-717b-480c-ab5a-c1d81658a00e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.370241 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.370150 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:16.647724 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.647700 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb"] Oct 31 04:01:16.650410 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:01:16.650381 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21db5238_717b_480c_ab5a_c1d81658a00e.slice/crio-3b81a1a9a168fc0640dbb2ee2a13fc121951c60b9b0bcf779360c5e7c32ca392 WatchSource:0}: Error finding container 3b81a1a9a168fc0640dbb2ee2a13fc121951c60b9b0bcf779360c5e7c32ca392: Status 404 returned error can't find the container with id 3b81a1a9a168fc0640dbb2ee2a13fc121951c60b9b0bcf779360c5e7c32ca392 Oct 31 04:01:16.653050 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.653010 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 04:01:16.653179 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.653084 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 04:01:16.653179 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:16.653116 5520 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 31 04:01:17.198726 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:17.198691 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" event={"ID":"21db5238-717b-480c-ab5a-c1d81658a00e","Type":"ContainerStarted","Data":"976e484d88d8ee43b412bb552595c13fb5c6f2750bed24f4dbaf7f91365bbf81"} Oct 31 04:01:17.199153 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:17.198748 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" event={"ID":"21db5238-717b-480c-ab5a-c1d81658a00e","Type":"ContainerStarted","Data":"3b81a1a9a168fc0640dbb2ee2a13fc121951c60b9b0bcf779360c5e7c32ca392"} Oct 31 04:01:17.220853 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:17.220796 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podStartSLOduration=1.220777133 podStartE2EDuration="1.220777133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:17.22015773 +0000 UTC m=+2701.414680862" watchObservedRunningTime="2026-10-31 04:01:17.220777133 +0000 UTC m=+2701.415300256" Oct 31 04:01:17.376664 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:17.376581 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:17.386687 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:17.386348 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 04:01:17.386687 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:17.386411 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 04:01:18.388230 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:18.380642 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 04:01:18.388230 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:18.380704 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 04:01:19.387230 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:19.380634 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 04:01:19.387230 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:19.380700 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 04:01:20.392363 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:20.392323 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 04:01:20.392813 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:20.392396 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 04:01:21.373235 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:21.372609 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 04:01:21.373235 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:21.372688 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 04:01:22.385229 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:22.381668 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 04:01:22.385229 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:22.381735 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 04:01:23.371486 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:23.371447 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 04:01:23.371663 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:23.371499 5520 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 04:01:24.815710 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:24.815677 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:24.816258 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:24.815997 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:24.816992 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:24.816972 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" Oct 31 04:01:39.357001 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:01:39.356918 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:01:50.675486 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.675446 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc"] Oct 31 04:01:50.678616 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.678600 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.685473 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.685442 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:01:50.685473 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.685456 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:01:50.685661 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.685506 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:01:50.685661 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.685457 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:01:50.687407 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.687386 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc"] Oct 31 04:01:50.830951 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.830915 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.831152 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.830981 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.831152 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.831092 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdx9\" (UniqueName: \"kubernetes.io/projected/b34fc71f-167e-4478-9f38-ebb37d5d33c7-kube-api-access-gkdx9\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.932238 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.932175 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.932431 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.932281 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.932431 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.932313 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdx9\" (UniqueName: \"kubernetes.io/projected/b34fc71f-167e-4478-9f38-ebb37d5d33c7-kube-api-access-gkdx9\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.933036 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.933008 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.934860 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.934837 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.940359 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.940336 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdx9\" (UniqueName: \"kubernetes.io/projected/b34fc71f-167e-4478-9f38-ebb37d5d33c7-kube-api-access-gkdx9\") pod \"maas-api-revoke-sub-c1e271a9-szchc\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:50.992834 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:50.992804 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:51.264556 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:51.264528 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc"] Oct 31 04:01:51.266624 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:01:51.266595 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34fc71f_167e_4478_9f38_ebb37d5d33c7.slice/crio-fb147c6507fddd509ec0b95fb251675c1e73b12bafca6f7bb265f327a9b08ca1 WatchSource:0}: Error finding container fb147c6507fddd509ec0b95fb251675c1e73b12bafca6f7bb265f327a9b08ca1: Status 404 returned error can't find the container with id fb147c6507fddd509ec0b95fb251675c1e73b12bafca6f7bb265f327a9b08ca1 Oct 31 04:01:51.418298 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:51.418258 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" event={"ID":"b34fc71f-167e-4478-9f38-ebb37d5d33c7","Type":"ContainerStarted","Data":"fb147c6507fddd509ec0b95fb251675c1e73b12bafca6f7bb265f327a9b08ca1"} Oct 31 04:01:52.424380 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:52.424287 5520 generic.go:356] "Generic (PLEG): container finished" podID="b34fc71f-167e-4478-9f38-ebb37d5d33c7" containerID="dfda0c0c0fbf08ae4cdfe67fb8a29e18f5c81197954d44e62fbcea33134695d1" exitCode=0 Oct 31 04:01:52.424380 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:52.424350 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" event={"ID":"b34fc71f-167e-4478-9f38-ebb37d5d33c7","Type":"ContainerDied","Data":"dfda0c0c0fbf08ae4cdfe67fb8a29e18f5c81197954d44e62fbcea33134695d1"} Oct 31 04:01:53.561328 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.561305 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:53.657996 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.657964 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-xks-ca\") pod \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " Oct 31 04:01:53.658246 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.658033 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34fc71f-167e-4478-9f38-ebb37d5d33c7-kube-api-access-gkdx9\" (UniqueName: \"kubernetes.io/projected/b34fc71f-167e-4478-9f38-ebb37d5d33c7-kube-api-access-gkdx9\") pod \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " Oct 31 04:01:53.658246 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.658085 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-openshift-ca\") pod \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\" (UID: \"b34fc71f-167e-4478-9f38-ebb37d5d33c7\") " Oct 31 04:01:53.658507 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.658481 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-openshift-ca" pod "b34fc71f-167e-4478-9f38-ebb37d5d33c7" (UID: "b34fc71f-167e-4478-9f38-ebb37d5d33c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:01:53.660454 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.660427 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-xks-ca" pod "b34fc71f-167e-4478-9f38-ebb37d5d33c7" (UID: "b34fc71f-167e-4478-9f38-ebb37d5d33c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:53.660565 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.660431 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34fc71f-167e-4478-9f38-ebb37d5d33c7-kube-api-access-gkdx9" pod "b34fc71f-167e-4478-9f38-ebb37d5d33c7" (UID: "b34fc71f-167e-4478-9f38-ebb37d5d33c7"). InnerVolumeSpecName "kube-api-access-gkdx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:53.758972 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.758888 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:53.758972 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.758921 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkdx9\" (UniqueName: \"kubernetes.io/projected/b34fc71f-167e-4478-9f38-ebb37d5d33c7-kube-api-access-gkdx9\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:53.758972 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:53.758932 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34fc71f-167e-4478-9f38-ebb37d5d33c7-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:54.438605 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:54.438571 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" Oct 31 04:01:54.438784 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:54.438574 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc" event={"ID":"b34fc71f-167e-4478-9f38-ebb37d5d33c7","Type":"ContainerDied","Data":"fb147c6507fddd509ec0b95fb251675c1e73b12bafca6f7bb265f327a9b08ca1"} Oct 31 04:01:54.438784 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:01:54.438708 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb147c6507fddd509ec0b95fb251675c1e73b12bafca6f7bb265f327a9b08ca1" Oct 31 04:02:06.383358 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:06.383313 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:02:06.384011 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:06.383978 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:02:06.420642 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:06.420302 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:02:06.420642 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:06.420375 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:02:09.357609 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:02:09.357573 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:02:09.406769 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.406735 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz"] Oct 31 04:02:09.407234 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.407215 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b34fc71f-167e-4478-9f38-ebb37d5d33c7" containerName="revoke-keys" Oct 31 04:02:09.407338 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.407236 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34fc71f-167e-4478-9f38-ebb37d5d33c7" containerName="revoke-keys" Oct 31 04:02:09.407390 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.407364 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34fc71f-167e-4478-9f38-ebb37d5d33c7" containerName="revoke-keys" Oct 31 04:02:09.410585 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.410565 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.418097 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.416548 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:02:09.418097 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.416591 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:02:09.418097 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.416625 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:02:09.418097 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.416869 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:02:09.419552 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.419526 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz"] Oct 31 04:02:09.458535 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.458502 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.458699 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.458539 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9622b\" (UniqueName: \"kubernetes.io/projected/e42fcc59-5750-4f01-89ad-f13f9715fad6-kube-api-access-9622b\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.458745 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.458700 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.560157 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.560124 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.560399 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.560178 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.560399 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.560229 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9622b\" (UniqueName: \"kubernetes.io/projected/e42fcc59-5750-4f01-89ad-f13f9715fad6-kube-api-access-9622b\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.560793 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.560769 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.562692 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.562668 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.568331 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.568307 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9622b\" (UniqueName: \"kubernetes.io/projected/e42fcc59-5750-4f01-89ad-f13f9715fad6-kube-api-access-9622b\") pod \"maas-api-revoke-sub-2e1f5b5f-g4nfz\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:09.730030 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:09.729993 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:10.001582 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.001551 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz"] Oct 31 04:02:10.004256 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:02:10.004232 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42fcc59_5750_4f01_89ad_f13f9715fad6.slice/crio-6f0e490ea0ba5345a6e8f543d8cb7bc1877ae7b8812d66cb16a42888c5303df0 WatchSource:0}: Error finding container 6f0e490ea0ba5345a6e8f543d8cb7bc1877ae7b8812d66cb16a42888c5303df0: Status 404 returned error can't find the container with id 6f0e490ea0ba5345a6e8f543d8cb7bc1877ae7b8812d66cb16a42888c5303df0 Oct 31 04:02:10.026014 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.025984 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf"] Oct 31 04:02:10.028093 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.028071 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2"] Oct 31 04:02:10.031977 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.031945 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c5211f-qwvkf"] Oct 31 04:02:10.032079 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.031985 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d859f302-lxdp2"] Oct 31 04:02:10.556098 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.555993 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" event={"ID":"e42fcc59-5750-4f01-89ad-f13f9715fad6","Type":"ContainerStarted","Data":"c5d3ed2c4af8ca9c5beea51078340aa30cc589458731139e90a50eeafe52f30a"} Oct 31 04:02:10.556098 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.556094 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" event={"ID":"e42fcc59-5750-4f01-89ad-f13f9715fad6","Type":"ContainerStarted","Data":"6f0e490ea0ba5345a6e8f543d8cb7bc1877ae7b8812d66cb16a42888c5303df0"} Oct 31 04:02:10.571562 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:10.571359 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" podStartSLOduration=1.571342403 podStartE2EDuration="1.571342403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:10.571171738 +0000 UTC m=+2754.765694860" watchObservedRunningTime="2026-10-31 04:02:10.571342403 +0000 UTC m=+2754.765865525" Oct 31 04:02:11.562027 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:11.561988 5520 generic.go:356] "Generic (PLEG): container finished" podID="e42fcc59-5750-4f01-89ad-f13f9715fad6" containerID="c5d3ed2c4af8ca9c5beea51078340aa30cc589458731139e90a50eeafe52f30a" exitCode=0 Oct 31 04:02:11.562541 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:11.562063 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" event={"ID":"e42fcc59-5750-4f01-89ad-f13f9715fad6","Type":"ContainerDied","Data":"c5d3ed2c4af8ca9c5beea51078340aa30cc589458731139e90a50eeafe52f30a"} Oct 31 04:02:11.915634 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:11.915556 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180074c2-4d96-4581-a301-a7e0e3b9ec27" path="/var/lib/kubelet/pods/180074c2-4d96-4581-a301-a7e0e3b9ec27/volumes" Oct 31 04:02:11.915970 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:11.915957 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdea8669-621e-4cae-8b36-d6057ba7955d" path="/var/lib/kubelet/pods/bdea8669-621e-4cae-8b36-d6057ba7955d/volumes" Oct 31 04:02:12.713903 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.713880 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:12.792686 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.792651 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42fcc59-5750-4f01-89ad-f13f9715fad6-kube-api-access-9622b\" (UniqueName: \"kubernetes.io/projected/e42fcc59-5750-4f01-89ad-f13f9715fad6-kube-api-access-9622b\") pod \"e42fcc59-5750-4f01-89ad-f13f9715fad6\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " Oct 31 04:02:12.792863 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.792728 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-xks-ca\") pod \"e42fcc59-5750-4f01-89ad-f13f9715fad6\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " Oct 31 04:02:12.792863 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.792779 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-openshift-ca\") pod \"e42fcc59-5750-4f01-89ad-f13f9715fad6\" (UID: \"e42fcc59-5750-4f01-89ad-f13f9715fad6\") " Oct 31 04:02:12.793223 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.793178 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-openshift-ca" pod "e42fcc59-5750-4f01-89ad-f13f9715fad6" (UID: "e42fcc59-5750-4f01-89ad-f13f9715fad6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:12.794951 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.794925 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-xks-ca" pod "e42fcc59-5750-4f01-89ad-f13f9715fad6" (UID: "e42fcc59-5750-4f01-89ad-f13f9715fad6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:12.795068 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.794953 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42fcc59-5750-4f01-89ad-f13f9715fad6-kube-api-access-9622b" pod "e42fcc59-5750-4f01-89ad-f13f9715fad6" (UID: "e42fcc59-5750-4f01-89ad-f13f9715fad6"). InnerVolumeSpecName "kube-api-access-9622b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:12.893612 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.893535 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9622b\" (UniqueName: \"kubernetes.io/projected/e42fcc59-5750-4f01-89ad-f13f9715fad6-kube-api-access-9622b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:12.893612 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.893564 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:12.893612 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:12.893575 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e42fcc59-5750-4f01-89ad-f13f9715fad6-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:13.573554 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:13.573494 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" Oct 31 04:02:13.573554 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:13.573519 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz" event={"ID":"e42fcc59-5750-4f01-89ad-f13f9715fad6","Type":"ContainerDied","Data":"6f0e490ea0ba5345a6e8f543d8cb7bc1877ae7b8812d66cb16a42888c5303df0"} Oct 31 04:02:13.573784 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:13.573576 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f0e490ea0ba5345a6e8f543d8cb7bc1877ae7b8812d66cb16a42888c5303df0" Oct 31 04:02:20.392945 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.392910 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p"] Oct 31 04:02:20.393493 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.393360 5520 cpu_manager.go:414] "containerMap: removing container" podUID="e42fcc59-5750-4f01-89ad-f13f9715fad6" containerName="revoke-keys" Oct 31 04:02:20.393493 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.393374 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42fcc59-5750-4f01-89ad-f13f9715fad6" containerName="revoke-keys" Oct 31 04:02:20.393493 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.393448 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42fcc59-5750-4f01-89ad-f13f9715fad6" containerName="revoke-keys" Oct 31 04:02:20.396962 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.396926 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.404400 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.404235 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:02:20.404400 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.404371 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:02:20.409192 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.408047 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:02:20.409192 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.408189 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:02:20.409554 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.409529 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p"] Oct 31 04:02:20.465462 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.465421 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxk92\" (UniqueName: \"kubernetes.io/projected/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-kube-api-access-rxk92\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.465643 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.465479 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.465643 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.465552 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.566722 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.566682 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxk92\" (UniqueName: \"kubernetes.io/projected/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-kube-api-access-rxk92\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.566722 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.566729 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.566951 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.566745 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.567604 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.567578 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.569358 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.569334 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.574603 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.574576 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxk92\" (UniqueName: \"kubernetes.io/projected/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-kube-api-access-rxk92\") pod \"maas-api-revoke-sub-ec843018-2p45p\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:20.714126 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:20.714092 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:21.035179 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:21.035149 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p"] Oct 31 04:02:21.037607 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:02:21.037578 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4dbcf1_394f_4a48_9668_c22d2fa51f49.slice/crio-b85d54f16fcecb675afcaeafebea07eec018b080f144b5dc2f788de5844d926b WatchSource:0}: Error finding container b85d54f16fcecb675afcaeafebea07eec018b080f144b5dc2f788de5844d926b: Status 404 returned error can't find the container with id b85d54f16fcecb675afcaeafebea07eec018b080f144b5dc2f788de5844d926b Oct 31 04:02:21.617376 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:21.617333 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" event={"ID":"bc4dbcf1-394f-4a48-9668-c22d2fa51f49","Type":"ContainerStarted","Data":"b0b3f3ccd9f4e2ea719dd59aac96c8f7d82a13f6d11c3d696bc20998c2b435b3"} Oct 31 04:02:21.617802 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:21.617409 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" event={"ID":"bc4dbcf1-394f-4a48-9668-c22d2fa51f49","Type":"ContainerStarted","Data":"b85d54f16fcecb675afcaeafebea07eec018b080f144b5dc2f788de5844d926b"} Oct 31 04:02:21.635098 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:21.635038 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" podStartSLOduration=1.6350238670000001 podStartE2EDuration="1.635023867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:21.632274909 +0000 UTC m=+2765.826798051" watchObservedRunningTime="2026-10-31 04:02:21.635023867 +0000 UTC m=+2765.829546987" Oct 31 04:02:22.628647 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:22.628605 5520 generic.go:356] "Generic (PLEG): container finished" podID="bc4dbcf1-394f-4a48-9668-c22d2fa51f49" containerID="b0b3f3ccd9f4e2ea719dd59aac96c8f7d82a13f6d11c3d696bc20998c2b435b3" exitCode=0 Oct 31 04:02:22.629162 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:22.628742 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" event={"ID":"bc4dbcf1-394f-4a48-9668-c22d2fa51f49","Type":"ContainerDied","Data":"b0b3f3ccd9f4e2ea719dd59aac96c8f7d82a13f6d11c3d696bc20998c2b435b3"} Oct 31 04:02:24.596228 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.593546 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:24.673402 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.671870 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" event={"ID":"bc4dbcf1-394f-4a48-9668-c22d2fa51f49","Type":"ContainerDied","Data":"b85d54f16fcecb675afcaeafebea07eec018b080f144b5dc2f788de5844d926b"} Oct 31 04:02:24.673800 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.673688 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b85d54f16fcecb675afcaeafebea07eec018b080f144b5dc2f788de5844d926b" Oct 31 04:02:24.674004 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.673939 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p" Oct 31 04:02:24.770314 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.761490 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-kube-api-access-rxk92\" (UniqueName: \"kubernetes.io/projected/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-kube-api-access-rxk92\") pod \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " Oct 31 04:02:24.770314 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.761581 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-xks-ca\") pod \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " Oct 31 04:02:24.770314 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.761660 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-openshift-ca\") pod \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\" (UID: \"bc4dbcf1-394f-4a48-9668-c22d2fa51f49\") " Oct 31 04:02:24.771693 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.771519 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-openshift-ca" pod "bc4dbcf1-394f-4a48-9668-c22d2fa51f49" (UID: "bc4dbcf1-394f-4a48-9668-c22d2fa51f49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:02:24.824374 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.824335 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-xks-ca" pod "bc4dbcf1-394f-4a48-9668-c22d2fa51f49" (UID: "bc4dbcf1-394f-4a48-9668-c22d2fa51f49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:02:24.827967 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.827937 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-kube-api-access-rxk92" pod "bc4dbcf1-394f-4a48-9668-c22d2fa51f49" (UID: "bc4dbcf1-394f-4a48-9668-c22d2fa51f49"). InnerVolumeSpecName "kube-api-access-rxk92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:02:24.863833 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.863792 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxk92\" (UniqueName: \"kubernetes.io/projected/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-kube-api-access-rxk92\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:24.864074 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.864055 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:24.864195 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:24.864180 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4dbcf1-394f-4a48-9668-c22d2fa51f49-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:02:27.018564 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:27.018530 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f"] Oct 31 04:02:27.020507 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:27.020477 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33abedb2-mkb8f"] Oct 31 04:02:27.915020 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:27.914984 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7308a1b0-065f-46f8-b369-1f27972b5b2c" path="/var/lib/kubelet/pods/7308a1b0-065f-46f8-b369-1f27972b5b2c/volumes" Oct 31 04:02:36.378227 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:36.377455 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5wkxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:02:36.378227 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:36.377573 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5wkxx" podUID="739ca5c0-6636-4c97-9236-04a5af2564d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:02:36.419592 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:36.419555 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f6mgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:02:36.419818 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:02:36.419649 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f6mgl" podUID="91077eb7-9c1a-4363-bdbf-e65fac8395d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:02:39.357768 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:02:39.357730 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:03:09.358320 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:03:09.358222 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:03:18.850941 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:18.850910 5520 scope.go:122] "RemoveContainer" containerID="160800bd78bd5ac7549117d25b9e54e57dbca0f68ca1a7e7f8bc638f2a1d99f5" Oct 31 04:03:18.859592 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:18.859569 5520 scope.go:122] "RemoveContainer" containerID="ed75e0299664dc04f3f15cc52322647cacaadc1da771e0d50320506f93084cef" Oct 31 04:03:34.022853 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:34.022808 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q"] Oct 31 04:03:34.026870 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:34.026842 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfc873c-r9h5q"] Oct 31 04:03:35.914929 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:35.914896 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efab1f60-b8e4-470a-b3c9-7212d5eb1ba1" path="/var/lib/kubelet/pods/efab1f60-b8e4-470a-b3c9-7212d5eb1ba1/volumes" Oct 31 04:03:39.019806 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:39.019771 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj"] Oct 31 04:03:39.022775 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:39.022744 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f309a45b-jbktj"] Oct 31 04:03:39.357575 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:03:39.357487 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:03:39.915341 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:39.915306 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d899d87-d746-4878-b5e0-4b7c30199599" path="/var/lib/kubelet/pods/6d899d87-d746-4878-b5e0-4b7c30199599/volumes" Oct 31 04:03:44.044931 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:44.044899 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6"] Oct 31 04:03:44.051359 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:44.051332 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8df7f75-z5dm6"] Oct 31 04:03:45.915273 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:45.915238 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac60650-0f41-4ee2-9f17-22435a8897b2" path="/var/lib/kubelet/pods/7ac60650-0f41-4ee2-9f17-22435a8897b2/volumes" Oct 31 04:03:49.019342 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:49.019304 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8"] Oct 31 04:03:49.021969 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:49.021936 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c94256-b49n8"] Oct 31 04:03:49.918429 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:03:49.918372 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9e2fdf-8b62-44bb-aec6-7785ff8cd80c" path="/var/lib/kubelet/pods/de9e2fdf-8b62-44bb-aec6-7785ff8cd80c/volumes" Oct 31 04:04:00.022844 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:00.022807 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk"] Oct 31 04:04:00.026780 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:00.026749 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd145a61-fwxtk"] Oct 31 04:04:01.915476 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:01.915445 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0edb99-c775-4d3a-a521-56f3e339d2d7" path="/var/lib/kubelet/pods/3e0edb99-c775-4d3a-a521-56f3e339d2d7/volumes" Oct 31 04:04:09.357879 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:04:09.357837 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:04:10.032297 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:10.032252 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f"] Oct 31 04:04:10.034504 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:10.034474 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4baf638-7rj9f"] Oct 31 04:04:11.941697 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:11.941650 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e43a043-741e-470f-ac12-2891ece53a6b" path="/var/lib/kubelet/pods/2e43a043-741e-470f-ac12-2891ece53a6b/volumes" Oct 31 04:04:18.882321 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:18.882288 5520 scope.go:122] "RemoveContainer" containerID="491ea8d1192fc00676d5e8fe15e7dfd10f55ae6dcc756cf1f981645280394e2f" Oct 31 04:04:39.496752 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:04:39.496543 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:04:54.025803 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:54.025761 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh"] Oct 31 04:04:54.028388 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:54.028360 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915e8093-ff5hh"] Oct 31 04:04:55.916136 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:04:55.916103 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f01cb39-fe54-46d5-a39e-6e379548da76" path="/var/lib/kubelet/pods/6f01cb39-fe54-46d5-a39e-6e379548da76/volumes" Oct 31 04:05:09.449439 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:05:09.449394 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:05:10.818334 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:10.818298 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:05:10.821303 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:10.818893 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:05:10.863357 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:10.862980 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:05:10.863357 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:10.863050 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:05:14.019403 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:14.019359 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx"] Oct 31 04:05:14.023950 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:14.023925 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-232b27ae-f76lx"] Oct 31 04:05:15.914523 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:15.914478 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b1acaf-1867-4e02-a7fa-e40944707568" path="/var/lib/kubelet/pods/c8b1acaf-1867-4e02-a7fa-e40944707568/volumes" Oct 31 04:05:18.905131 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.905100 5520 scope.go:122] "RemoveContainer" containerID="2bdf556214ef23fec943adb6ca83d73543749daa213ec42fed6b54fee2e97153" Oct 31 04:05:18.913838 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.913813 5520 scope.go:122] "RemoveContainer" containerID="a5a15a05904f968ab9104c2f8661e9b463da6c3673bd9020932a722bc71ed9c5" Oct 31 04:05:18.921971 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.921946 5520 scope.go:122] "RemoveContainer" containerID="fda761e8e35c0cfddea5bc25ff73ab8d24085b4e88abb3e78d3e50be0de5e159" Oct 31 04:05:18.930503 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.930424 5520 scope.go:122] "RemoveContainer" containerID="1e87ff301c3fe98543579b0d3d8912403a6d4daf316741d4a7133224a2b48d0b" Oct 31 04:05:18.940650 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.940620 5520 scope.go:122] "RemoveContainer" containerID="27c01bb25b864d061db7c395636a96097912f430db93f56c61b55a91b85c6b14" Oct 31 04:05:18.948223 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.948191 5520 scope.go:122] "RemoveContainer" containerID="16390e848426f107409014aca9d1a8b187732eb822e33ddc19e02b689be30f7b" Oct 31 04:05:18.955943 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.955926 5520 scope.go:122] "RemoveContainer" containerID="44b9dc54ed9ff7253340d1e2b24370cb1aaced0b4070709b8071070e86a9ac90" Oct 31 04:05:18.963043 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.963017 5520 scope.go:122] "RemoveContainer" containerID="9675eed83c09d42d700e88dc7847fefb9aa497e704547e862d4dca630e6c5b4b" Oct 31 04:05:18.970404 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:18.970385 5520 scope.go:122] "RemoveContainer" containerID="cb60710623cfda4ebfe4f19a61da597bf45e2c556922db3e727e8a28c07c4b19" Oct 31 04:05:39.357113 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:05:39.357072 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:05:59.020386 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:59.020305 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs"] Oct 31 04:05:59.022956 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:59.022925 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-p74cs"] Oct 31 04:05:59.918344 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:05:59.918308 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f16c06-8bcb-43e9-aef5-0bcde3c18356" path="/var/lib/kubelet/pods/14f16c06-8bcb-43e9-aef5-0bcde3c18356/volumes" Oct 31 04:06:09.356963 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:06:09.356903 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:06:19.033137 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:19.033087 5520 scope.go:122] "RemoveContainer" containerID="bed900e71e71c5ff629f710a533d12efd9ed3dd33aa9af9e4958052fe764360f" Oct 31 04:06:19.046435 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:19.046320 5520 scope.go:122] "RemoveContainer" containerID="280e89d0008a738c2f906cece273c5ac4cecdcae7a7718d50f592f074a175b5c" Oct 31 04:06:19.055456 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:19.055428 5520 scope.go:122] "RemoveContainer" containerID="051a8f2c0b4afadadddd0804c34056893852fa62a69d737918fc8ff5387f8761" Oct 31 04:06:19.063119 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:19.063099 5520 scope.go:122] "RemoveContainer" containerID="a4e724710cf2f56ab0b4ea6140e4b2cbbf5d9a451f2d0d40dc2253f06ae13a56" Oct 31 04:06:39.357417 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:06:39.357377 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:06:44.111948 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.111917 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5"] Oct 31 04:06:44.112372 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.112303 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bc4dbcf1-394f-4a48-9668-c22d2fa51f49" containerName="revoke-keys" Oct 31 04:06:44.112372 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.112315 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4dbcf1-394f-4a48-9668-c22d2fa51f49" containerName="revoke-keys" Oct 31 04:06:44.112372 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.112372 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4dbcf1-394f-4a48-9668-c22d2fa51f49" containerName="revoke-keys" Oct 31 04:06:44.117460 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.117439 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.123173 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.123145 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:44.123359 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.123235 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:44.123359 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.123311 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:06:44.123359 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.123315 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:44.124402 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.124382 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5"] Oct 31 04:06:44.245506 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.245463 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsvg\" (UniqueName: \"kubernetes.io/projected/32b08e78-5e6b-4a0a-a0d8-afa25756816c-kube-api-access-8hsvg\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.245506 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.245509 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.245751 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.245635 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.346697 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.346663 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsvg\" (UniqueName: \"kubernetes.io/projected/32b08e78-5e6b-4a0a-a0d8-afa25756816c-kube-api-access-8hsvg\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.346697 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.346707 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.346983 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.346763 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.347805 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.347774 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.349399 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.349376 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.355858 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.355835 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsvg\" (UniqueName: \"kubernetes.io/projected/32b08e78-5e6b-4a0a-a0d8-afa25756816c-kube-api-access-8hsvg\") pod \"maas-api-revoke-sub-8447e295-kcch5\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.432512 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.432476 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:44.703539 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:44.703516 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5"] Oct 31 04:06:44.706423 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:06:44.706393 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b08e78_5e6b_4a0a_a0d8_afa25756816c.slice/crio-d42a8442b40714f8f3157edb8b3ec075a43582d05502fad87b975254ee958f36 WatchSource:0}: Error finding container d42a8442b40714f8f3157edb8b3ec075a43582d05502fad87b975254ee958f36: Status 404 returned error can't find the container with id d42a8442b40714f8f3157edb8b3ec075a43582d05502fad87b975254ee958f36 Oct 31 04:06:45.361374 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:45.361271 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" event={"ID":"32b08e78-5e6b-4a0a-a0d8-afa25756816c","Type":"ContainerStarted","Data":"ab713b6524fade17aed28a5adca92fd96e940ae5d6f7b5f97f4c6d91bf801aeb"} Oct 31 04:06:45.361374 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:45.361339 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" event={"ID":"32b08e78-5e6b-4a0a-a0d8-afa25756816c","Type":"ContainerStarted","Data":"d42a8442b40714f8f3157edb8b3ec075a43582d05502fad87b975254ee958f36"} Oct 31 04:06:45.379092 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:45.379036 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" podStartSLOduration=1.379021565 podStartE2EDuration="1.379021565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:45.377303003 +0000 UTC m=+3029.571826135" watchObservedRunningTime="2026-10-31 04:06:45.379021565 +0000 UTC m=+3029.573544679" Oct 31 04:06:46.367324 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:46.367286 5520 generic.go:356] "Generic (PLEG): container finished" podID="32b08e78-5e6b-4a0a-a0d8-afa25756816c" containerID="ab713b6524fade17aed28a5adca92fd96e940ae5d6f7b5f97f4c6d91bf801aeb" exitCode=0 Oct 31 04:06:46.367788 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:46.367349 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" event={"ID":"32b08e78-5e6b-4a0a-a0d8-afa25756816c","Type":"ContainerDied","Data":"ab713b6524fade17aed28a5adca92fd96e940ae5d6f7b5f97f4c6d91bf801aeb"} Oct 31 04:06:47.496669 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.496643 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:47.580465 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.580432 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b08e78-5e6b-4a0a-a0d8-afa25756816c-kube-api-access-8hsvg\" (UniqueName: \"kubernetes.io/projected/32b08e78-5e6b-4a0a-a0d8-afa25756816c-kube-api-access-8hsvg\") pod \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " Oct 31 04:06:47.580636 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.580473 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-xks-ca\") pod \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " Oct 31 04:06:47.580636 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.580572 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-openshift-ca\") pod \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\" (UID: \"32b08e78-5e6b-4a0a-a0d8-afa25756816c\") " Oct 31 04:06:47.580936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.580907 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-openshift-ca" pod "32b08e78-5e6b-4a0a-a0d8-afa25756816c" (UID: "32b08e78-5e6b-4a0a-a0d8-afa25756816c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:47.582795 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.582774 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-xks-ca" pod "32b08e78-5e6b-4a0a-a0d8-afa25756816c" (UID: "32b08e78-5e6b-4a0a-a0d8-afa25756816c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:47.582903 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.582859 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b08e78-5e6b-4a0a-a0d8-afa25756816c-kube-api-access-8hsvg" pod "32b08e78-5e6b-4a0a-a0d8-afa25756816c" (UID: "32b08e78-5e6b-4a0a-a0d8-afa25756816c"). InnerVolumeSpecName "kube-api-access-8hsvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:47.681223 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.681172 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:47.681394 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.681225 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hsvg\" (UniqueName: \"kubernetes.io/projected/32b08e78-5e6b-4a0a-a0d8-afa25756816c-kube-api-access-8hsvg\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:47.681394 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:47.681252 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b08e78-5e6b-4a0a-a0d8-afa25756816c-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:48.377289 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:48.377255 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" event={"ID":"32b08e78-5e6b-4a0a-a0d8-afa25756816c","Type":"ContainerDied","Data":"d42a8442b40714f8f3157edb8b3ec075a43582d05502fad87b975254ee958f36"} Oct 31 04:06:48.377289 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:48.377277 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5" Oct 31 04:06:48.377485 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:48.377304 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d42a8442b40714f8f3157edb8b3ec075a43582d05502fad87b975254ee958f36" Oct 31 04:06:54.020326 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:54.020290 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc"] Oct 31 04:06:54.022561 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:54.022534 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-szchc"] Oct 31 04:06:55.125549 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.125512 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f"] Oct 31 04:06:55.125961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.125881 5520 cpu_manager.go:414] "containerMap: removing container" podUID="32b08e78-5e6b-4a0a-a0d8-afa25756816c" containerName="revoke-keys" Oct 31 04:06:55.125961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.125892 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b08e78-5e6b-4a0a-a0d8-afa25756816c" containerName="revoke-keys" Oct 31 04:06:55.125961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.125960 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b08e78-5e6b-4a0a-a0d8-afa25756816c" containerName="revoke-keys" Oct 31 04:06:55.129152 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.129132 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.135340 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.135313 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:06:55.136251 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.135807 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:06:55.136251 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.135838 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:06:55.136251 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.135913 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:06:55.137936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.137792 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f"] Oct 31 04:06:55.272525 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.272475 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4zn\" (UniqueName: \"kubernetes.io/projected/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-kube-api-access-lm4zn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.272525 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.272527 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.272819 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.272689 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.374212 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.374152 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.374430 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.374275 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4zn\" (UniqueName: \"kubernetes.io/projected/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-kube-api-access-lm4zn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.374430 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.374311 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.374830 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.374796 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.376937 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.376880 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.382190 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.382166 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4zn\" (UniqueName: \"kubernetes.io/projected/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-kube-api-access-lm4zn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.442748 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.442712 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:55.723140 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.723108 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f"] Oct 31 04:06:55.727155 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:06:55.727114 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1f2f89_ae88_4efa_a5a5_a07f39411ae3.slice/crio-65f29d1f93ab10bb28911baf269fe8d7f98c1c1d90a158423788a7aa99718f98 WatchSource:0}: Error finding container 65f29d1f93ab10bb28911baf269fe8d7f98c1c1d90a158423788a7aa99718f98: Status 404 returned error can't find the container with id 65f29d1f93ab10bb28911baf269fe8d7f98c1c1d90a158423788a7aa99718f98 Oct 31 04:06:55.914986 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:55.914953 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34fc71f-167e-4478-9f38-ebb37d5d33c7" path="/var/lib/kubelet/pods/b34fc71f-167e-4478-9f38-ebb37d5d33c7/volumes" Oct 31 04:06:56.413260 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:56.413196 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" event={"ID":"de1f2f89-ae88-4efa-a5a5-a07f39411ae3","Type":"ContainerStarted","Data":"f9dbec417ab2bcee8f555ea67353b678ba884090af7b5397ada2128186aedfac"} Oct 31 04:06:56.413676 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:56.413276 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" event={"ID":"de1f2f89-ae88-4efa-a5a5-a07f39411ae3","Type":"ContainerStarted","Data":"65f29d1f93ab10bb28911baf269fe8d7f98c1c1d90a158423788a7aa99718f98"} Oct 31 04:06:56.433096 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:56.433041 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" podStartSLOduration=1.433026664 podStartE2EDuration="1.433026664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:56.432245204 +0000 UTC m=+3040.626768318" watchObservedRunningTime="2026-10-31 04:06:56.433026664 +0000 UTC m=+3040.627549784" Oct 31 04:06:57.419327 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:57.419288 5520 generic.go:356] "Generic (PLEG): container finished" podID="de1f2f89-ae88-4efa-a5a5-a07f39411ae3" containerID="f9dbec417ab2bcee8f555ea67353b678ba884090af7b5397ada2128186aedfac" exitCode=0 Oct 31 04:06:57.419327 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:57.419326 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" event={"ID":"de1f2f89-ae88-4efa-a5a5-a07f39411ae3","Type":"ContainerDied","Data":"f9dbec417ab2bcee8f555ea67353b678ba884090af7b5397ada2128186aedfac"} Oct 31 04:06:58.559057 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.559027 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:58.715998 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.715958 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-xks-ca\") pod \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " Oct 31 04:06:58.716183 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.716008 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-openshift-ca\") pod \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " Oct 31 04:06:58.716183 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.716108 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-kube-api-access-lm4zn\" (UniqueName: \"kubernetes.io/projected/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-kube-api-access-lm4zn\") pod \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\" (UID: \"de1f2f89-ae88-4efa-a5a5-a07f39411ae3\") " Oct 31 04:06:58.716501 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.716468 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-openshift-ca" pod "de1f2f89-ae88-4efa-a5a5-a07f39411ae3" (UID: "de1f2f89-ae88-4efa-a5a5-a07f39411ae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:06:58.718436 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.718414 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-kube-api-access-lm4zn" pod "de1f2f89-ae88-4efa-a5a5-a07f39411ae3" (UID: "de1f2f89-ae88-4efa-a5a5-a07f39411ae3"). InnerVolumeSpecName "kube-api-access-lm4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:06:58.718436 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.718416 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-xks-ca" pod "de1f2f89-ae88-4efa-a5a5-a07f39411ae3" (UID: "de1f2f89-ae88-4efa-a5a5-a07f39411ae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:06:58.817219 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.817178 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:58.817388 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.817235 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:58.817388 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:58.817247 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4zn\" (UniqueName: \"kubernetes.io/projected/de1f2f89-ae88-4efa-a5a5-a07f39411ae3-kube-api-access-lm4zn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:06:59.429128 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:59.429094 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" Oct 31 04:06:59.429352 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:59.429126 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f" event={"ID":"de1f2f89-ae88-4efa-a5a5-a07f39411ae3","Type":"ContainerDied","Data":"65f29d1f93ab10bb28911baf269fe8d7f98c1c1d90a158423788a7aa99718f98"} Oct 31 04:06:59.429352 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:06:59.429176 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f29d1f93ab10bb28911baf269fe8d7f98c1c1d90a158423788a7aa99718f98" Oct 31 04:07:06.482424 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:06.482389 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f"] Oct 31 04:07:06.484748 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:06.484719 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-799962-54t9f"] Oct 31 04:07:07.917061 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:07.917016 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1f2f89-ae88-4efa-a5a5-a07f39411ae3" path="/var/lib/kubelet/pods/de1f2f89-ae88-4efa-a5a5-a07f39411ae3/volumes" Oct 31 04:07:09.357021 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:07:09.356984 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:07:13.024449 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:13.024412 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz"] Oct 31 04:07:13.026971 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:13.026947 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4nfz"] Oct 31 04:07:13.914615 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:13.914584 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42fcc59-5750-4f01-89ad-f13f9715fad6" path="/var/lib/kubelet/pods/e42fcc59-5750-4f01-89ad-f13f9715fad6/volumes" Oct 31 04:07:19.095048 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:19.094622 5520 scope.go:122] "RemoveContainer" containerID="d7d9fcded7a4412afa03f9f71a28480763d102ee70e1d074f0fc80601a33a27e" Oct 31 04:07:24.021758 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:24.021726 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p"] Oct 31 04:07:24.024302 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:24.024278 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2p45p"] Oct 31 04:07:26.008737 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:26.008643 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4dbcf1-394f-4a48-9668-c22d2fa51f49" path="/var/lib/kubelet/pods/bc4dbcf1-394f-4a48-9668-c22d2fa51f49/volumes" Oct 31 04:07:33.030491 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.030457 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8"] Oct 31 04:07:33.030957 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.030662 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" podUID="b9f94808-efa2-438d-8cf2-20a3c5f93e21" containerName="manager" containerID="cri-o://1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5" gracePeriod=10 Oct 31 04:07:33.280832 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.280771 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 04:07:33.413370 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.413332 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f94808-efa2-438d-8cf2-20a3c5f93e21-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9f94808-efa2-438d-8cf2-20a3c5f93e21-extensions-socket-volume\") pod \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " Oct 31 04:07:33.413577 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.413393 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f94808-efa2-438d-8cf2-20a3c5f93e21-kube-api-access-q24gk\" (UniqueName: \"kubernetes.io/projected/b9f94808-efa2-438d-8cf2-20a3c5f93e21-kube-api-access-q24gk\") pod \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\" (UID: \"b9f94808-efa2-438d-8cf2-20a3c5f93e21\") " Oct 31 04:07:33.413737 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.413711 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f94808-efa2-438d-8cf2-20a3c5f93e21-extensions-socket-volume" pod "b9f94808-efa2-438d-8cf2-20a3c5f93e21" (UID: "b9f94808-efa2-438d-8cf2-20a3c5f93e21"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:33.415607 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.415589 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f94808-efa2-438d-8cf2-20a3c5f93e21-kube-api-access-q24gk" pod "b9f94808-efa2-438d-8cf2-20a3c5f93e21" (UID: "b9f94808-efa2-438d-8cf2-20a3c5f93e21"). InnerVolumeSpecName "kube-api-access-q24gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:33.514785 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.514740 5520 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9f94808-efa2-438d-8cf2-20a3c5f93e21-extensions-socket-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:33.514785 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.514778 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q24gk\" (UniqueName: \"kubernetes.io/projected/b9f94808-efa2-438d-8cf2-20a3c5f93e21-kube-api-access-q24gk\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:33.596670 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.596582 5520 generic.go:356] "Generic (PLEG): container finished" podID="b9f94808-efa2-438d-8cf2-20a3c5f93e21" containerID="1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5" exitCode=0 Oct 31 04:07:33.596670 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.596635 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" Oct 31 04:07:33.596877 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.596671 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" event={"ID":"b9f94808-efa2-438d-8cf2-20a3c5f93e21","Type":"ContainerDied","Data":"1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5"} Oct 31 04:07:33.596877 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.596728 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8" event={"ID":"b9f94808-efa2-438d-8cf2-20a3c5f93e21","Type":"ContainerDied","Data":"89e9ba7f29b5db4c17ed77501e96545da08e559345a2ab5964b6d6ca498e1ef4"} Oct 31 04:07:33.596877 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.596750 5520 scope.go:122] "RemoveContainer" containerID="1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5" Oct 31 04:07:33.607786 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.607765 5520 scope.go:122] "RemoveContainer" containerID="1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5" Oct 31 04:07:33.608071 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:07:33.608052 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5\": container with ID starting with 1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5 not found: ID does not exist" containerID="1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5" Oct 31 04:07:33.608120 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.608081 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5"} err="failed to get container status \"1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5\": rpc error: code = NotFound desc = could not find container \"1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5\": container with ID starting with 1428bffd6f3611abf17ba45a99fcc457d5b3aa77296a1a3edb4cebab4e7605c5 not found: ID does not exist" Oct 31 04:07:33.621313 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.621293 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8"] Oct 31 04:07:33.624287 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.624267 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-654t8"] Oct 31 04:07:33.915386 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:07:33.915306 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f94808-efa2-438d-8cf2-20a3c5f93e21" path="/var/lib/kubelet/pods/b9f94808-efa2-438d-8cf2-20a3c5f93e21/volumes" Oct 31 04:07:39.359881 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:07:39.359840 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:08:09.357428 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:08:09.357383 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:08:19.120774 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:19.120739 5520 scope.go:122] "RemoveContainer" containerID="dfda0c0c0fbf08ae4cdfe67fb8a29e18f5c81197954d44e62fbcea33134695d1" Oct 31 04:08:19.129170 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:19.128969 5520 scope.go:122] "RemoveContainer" containerID="c5d3ed2c4af8ca9c5beea51078340aa30cc589458731139e90a50eeafe52f30a" Oct 31 04:08:39.358008 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:08:39.357967 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:08:56.378104 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.378016 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg"] Oct 31 04:08:56.378570 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.378386 5520 cpu_manager.go:414] "containerMap: removing container" podUID="b9f94808-efa2-438d-8cf2-20a3c5f93e21" containerName="manager" Oct 31 04:08:56.378570 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.378397 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f94808-efa2-438d-8cf2-20a3c5f93e21" containerName="manager" Oct 31 04:08:56.378570 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.378411 5520 cpu_manager.go:414] "containerMap: removing container" podUID="de1f2f89-ae88-4efa-a5a5-a07f39411ae3" containerName="revoke-keys" Oct 31 04:08:56.378570 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.378416 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1f2f89-ae88-4efa-a5a5-a07f39411ae3" containerName="revoke-keys" Oct 31 04:08:56.378570 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.378492 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f94808-efa2-438d-8cf2-20a3c5f93e21" containerName="manager" Oct 31 04:08:56.378570 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.378500 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1f2f89-ae88-4efa-a5a5-a07f39411ae3" containerName="revoke-keys" Oct 31 04:08:56.381598 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.381580 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.388493 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.388414 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:08:56.388746 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.388717 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:08:56.388861 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.388763 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:08:56.388861 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.388826 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:08:56.390270 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.390239 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg"] Oct 31 04:08:56.450303 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.450261 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.450303 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.450313 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.450556 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.450330 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmm8b\" (UniqueName: \"kubernetes.io/projected/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-kube-api-access-gmm8b\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.551701 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.551663 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.551701 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.551706 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.552003 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.551724 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmm8b\" (UniqueName: \"kubernetes.io/projected/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-kube-api-access-gmm8b\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.552437 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.552413 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.554323 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.554304 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.559752 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.559732 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmm8b\" (UniqueName: \"kubernetes.io/projected/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-kube-api-access-gmm8b\") pod \"maas-api-revoke-sub-d576e68b-w5vvg\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.695870 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.695837 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:56.966563 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:56.966370 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg"] Oct 31 04:08:56.969383 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:08:56.969353 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e88f23_618c_4c1f_a8ac_3f8dd36a20b1.slice/crio-3abbf70cadd53b1564f2eb0b028f4abf91e8cb2728b159254438b19ef9e27f71 WatchSource:0}: Error finding container 3abbf70cadd53b1564f2eb0b028f4abf91e8cb2728b159254438b19ef9e27f71: Status 404 returned error can't find the container with id 3abbf70cadd53b1564f2eb0b028f4abf91e8cb2728b159254438b19ef9e27f71 Oct 31 04:08:57.026072 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:57.026036 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" event={"ID":"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1","Type":"ContainerStarted","Data":"3abbf70cadd53b1564f2eb0b028f4abf91e8cb2728b159254438b19ef9e27f71"} Oct 31 04:08:58.034333 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:58.034226 5520 generic.go:356] "Generic (PLEG): container finished" podID="87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" containerID="b4baa350bb172b611d0e9b9e70cd5e5bec60461a0705694fb96c11686c5e7c1f" exitCode=0 Oct 31 04:08:58.034333 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:58.034262 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" event={"ID":"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1","Type":"ContainerDied","Data":"b4baa350bb172b611d0e9b9e70cd5e5bec60461a0705694fb96c11686c5e7c1f"} Oct 31 04:08:59.167403 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.167377 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:08:59.278163 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.278122 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-xks-ca\") pod \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " Oct 31 04:08:59.278378 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.278228 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-kube-api-access-gmm8b\" (UniqueName: \"kubernetes.io/projected/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-kube-api-access-gmm8b\") pod \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " Oct 31 04:08:59.278378 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.278298 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-openshift-ca\") pod \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\" (UID: \"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1\") " Oct 31 04:08:59.278683 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.278662 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-openshift-ca" pod "87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" (UID: "87e88f23-618c-4c1f-a8ac-3f8dd36a20b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:08:59.280538 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.280514 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-kube-api-access-gmm8b" pod "87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" (UID: "87e88f23-618c-4c1f-a8ac-3f8dd36a20b1"). InnerVolumeSpecName "kube-api-access-gmm8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:08:59.280614 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.280521 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-xks-ca" pod "87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" (UID: "87e88f23-618c-4c1f-a8ac-3f8dd36a20b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:08:59.379992 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.379908 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:59.379992 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.379940 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:08:59.379992 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:08:59.379950 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmm8b\" (UniqueName: \"kubernetes.io/projected/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1-kube-api-access-gmm8b\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:00.044389 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:00.044352 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" Oct 31 04:09:00.044577 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:00.044358 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg" event={"ID":"87e88f23-618c-4c1f-a8ac-3f8dd36a20b1","Type":"ContainerDied","Data":"3abbf70cadd53b1564f2eb0b028f4abf91e8cb2728b159254438b19ef9e27f71"} Oct 31 04:09:00.044577 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:00.044497 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3abbf70cadd53b1564f2eb0b028f4abf91e8cb2728b159254438b19ef9e27f71" Oct 31 04:09:09.371981 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:09:09.371937 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:09:15.098832 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.098797 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md"] Oct 31 04:09:15.099282 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.099217 5520 cpu_manager.go:414] "containerMap: removing container" podUID="87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" containerName="revoke-keys" Oct 31 04:09:15.099282 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.099233 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" containerName="revoke-keys" Oct 31 04:09:15.099374 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.099307 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" containerName="revoke-keys" Oct 31 04:09:15.102545 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.102529 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.106665 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.106645 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:09:15.106665 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.106671 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:09:15.106836 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.106691 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:09:15.106836 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.106707 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:09:15.111298 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.111276 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md"] Oct 31 04:09:15.156414 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.156374 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgbn\" (UniqueName: \"kubernetes.io/projected/bd39867c-bb78-4c15-b88a-5576789791a0-kube-api-access-5lgbn\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.156617 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.156500 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.156617 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.156596 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.257702 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.257661 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.257876 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.257713 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgbn\" (UniqueName: \"kubernetes.io/projected/bd39867c-bb78-4c15-b88a-5576789791a0-kube-api-access-5lgbn\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.257876 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.257749 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.258341 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.258316 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.260175 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.260153 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.265594 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.265574 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgbn\" (UniqueName: \"kubernetes.io/projected/bd39867c-bb78-4c15-b88a-5576789791a0-kube-api-access-5lgbn\") pod \"maas-api-revoke-sub-d2bc93bc-gh5md\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.417383 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.417295 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:15.688357 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:15.688331 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md"] Oct 31 04:09:15.690675 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:09:15.690647 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39867c_bb78_4c15_b88a_5576789791a0.slice/crio-e3825b47f15c17e40ae091fd8d52eca86fe09804372fff9757d977fa2254316a WatchSource:0}: Error finding container e3825b47f15c17e40ae091fd8d52eca86fe09804372fff9757d977fa2254316a: Status 404 returned error can't find the container with id e3825b47f15c17e40ae091fd8d52eca86fe09804372fff9757d977fa2254316a Oct 31 04:09:16.124475 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:16.124439 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" event={"ID":"bd39867c-bb78-4c15-b88a-5576789791a0","Type":"ContainerStarted","Data":"d3af41c097c8215739a1ee2ea6c2a8dcb4877d343ac03151c1075b8a8bb02e8f"} Oct 31 04:09:16.124877 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:16.124492 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" event={"ID":"bd39867c-bb78-4c15-b88a-5576789791a0","Type":"ContainerStarted","Data":"e3825b47f15c17e40ae091fd8d52eca86fe09804372fff9757d977fa2254316a"} Oct 31 04:09:16.141043 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:16.140991 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" podStartSLOduration=1.140979564 podStartE2EDuration="1.140979564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:16.139788104 +0000 UTC m=+3180.334311242" watchObservedRunningTime="2026-10-31 04:09:16.140979564 +0000 UTC m=+3180.335502675" Oct 31 04:09:17.130189 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:17.130150 5520 generic.go:356] "Generic (PLEG): container finished" podID="bd39867c-bb78-4c15-b88a-5576789791a0" containerID="d3af41c097c8215739a1ee2ea6c2a8dcb4877d343ac03151c1075b8a8bb02e8f" exitCode=0 Oct 31 04:09:17.130617 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:17.130233 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" event={"ID":"bd39867c-bb78-4c15-b88a-5576789791a0","Type":"ContainerDied","Data":"d3af41c097c8215739a1ee2ea6c2a8dcb4877d343ac03151c1075b8a8bb02e8f"} Oct 31 04:09:18.267368 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.267340 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:18.388675 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.388580 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd39867c-bb78-4c15-b88a-5576789791a0-kube-api-access-5lgbn\" (UniqueName: \"kubernetes.io/projected/bd39867c-bb78-4c15-b88a-5576789791a0-kube-api-access-5lgbn\") pod \"bd39867c-bb78-4c15-b88a-5576789791a0\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " Oct 31 04:09:18.388675 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.388625 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-openshift-ca\") pod \"bd39867c-bb78-4c15-b88a-5576789791a0\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " Oct 31 04:09:18.388929 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.388686 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-xks-ca\") pod \"bd39867c-bb78-4c15-b88a-5576789791a0\" (UID: \"bd39867c-bb78-4c15-b88a-5576789791a0\") " Oct 31 04:09:18.388984 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.388928 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-openshift-ca" pod "bd39867c-bb78-4c15-b88a-5576789791a0" (UID: "bd39867c-bb78-4c15-b88a-5576789791a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:09:18.389107 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.389086 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:18.390908 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.390887 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-xks-ca" pod "bd39867c-bb78-4c15-b88a-5576789791a0" (UID: "bd39867c-bb78-4c15-b88a-5576789791a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:09:18.391001 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.390966 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd39867c-bb78-4c15-b88a-5576789791a0-kube-api-access-5lgbn" pod "bd39867c-bb78-4c15-b88a-5576789791a0" (UID: "bd39867c-bb78-4c15-b88a-5576789791a0"). InnerVolumeSpecName "kube-api-access-5lgbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:09:18.490439 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.490402 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgbn\" (UniqueName: \"kubernetes.io/projected/bd39867c-bb78-4c15-b88a-5576789791a0-kube-api-access-5lgbn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:18.490439 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:18.490433 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd39867c-bb78-4c15-b88a-5576789791a0-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:09:19.139843 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:19.139810 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" Oct 31 04:09:19.140003 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:19.139850 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md" event={"ID":"bd39867c-bb78-4c15-b88a-5576789791a0","Type":"ContainerDied","Data":"e3825b47f15c17e40ae091fd8d52eca86fe09804372fff9757d977fa2254316a"} Oct 31 04:09:19.140003 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:19.139896 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3825b47f15c17e40ae091fd8d52eca86fe09804372fff9757d977fa2254316a" Oct 31 04:09:19.153191 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:19.153164 5520 scope.go:122] "RemoveContainer" containerID="b0b3f3ccd9f4e2ea719dd59aac96c8f7d82a13f6d11c3d696bc20998c2b435b3" Oct 31 04:09:31.993792 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:31.993755 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5"] Oct 31 04:09:31.994311 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:31.994292 5520 cpu_manager.go:414] "containerMap: removing container" podUID="bd39867c-bb78-4c15-b88a-5576789791a0" containerName="revoke-keys" Oct 31 04:09:31.994365 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:31.994313 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd39867c-bb78-4c15-b88a-5576789791a0" containerName="revoke-keys" Oct 31 04:09:31.994402 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:31.994376 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd39867c-bb78-4c15-b88a-5576789791a0" containerName="revoke-keys" Oct 31 04:09:31.998891 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:31.998866 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.008111 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.008091 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8a336e\"" Oct 31 04:09:32.018836 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.018810 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5"] Oct 31 04:09:32.047789 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.047749 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8abb7425-dockercfg-ct2xs\"" Oct 31 04:09:32.118645 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.118607 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.118811 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.118768 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpnn\" (UniqueName: \"kubernetes.io/projected/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-kube-api-access-vrpnn\") pod \"payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.222224 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.221759 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpnn\" (UniqueName: \"kubernetes.io/projected/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-kube-api-access-vrpnn\") pod \"payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.222224 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.221830 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.222922 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.222887 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.232646 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.232617 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpnn\" (UniqueName: \"kubernetes.io/projected/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-kube-api-access-vrpnn\") pod \"payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.313123 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.313045 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:32.596458 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:32.596381 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5"] Oct 31 04:09:32.599537 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:09:32.599510 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7e466e_e1ee_439d_ac49_29ccb3bd7573.slice/crio-87f0b771924e728420d17c0925f2c299a93ced7143c580faef029bdc82ccb062 WatchSource:0}: Error finding container 87f0b771924e728420d17c0925f2c299a93ced7143c580faef029bdc82ccb062: Status 404 returned error can't find the container with id 87f0b771924e728420d17c0925f2c299a93ced7143c580faef029bdc82ccb062 Oct 31 04:09:33.220123 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:33.220079 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" event={"ID":"ac7e466e-e1ee-439d-ac49-29ccb3bd7573","Type":"ContainerStarted","Data":"f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190"} Oct 31 04:09:33.220619 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:33.220134 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" event={"ID":"ac7e466e-e1ee-439d-ac49-29ccb3bd7573","Type":"ContainerStarted","Data":"87f0b771924e728420d17c0925f2c299a93ced7143c580faef029bdc82ccb062"} Oct 31 04:09:33.220619 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:33.220294 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:33.236899 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:33.236856 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" podStartSLOduration=2.236837765 podStartE2EDuration="2.236837765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:33.236351887 +0000 UTC m=+3197.430875019" watchObservedRunningTime="2026-10-31 04:09:33.236837765 +0000 UTC m=+3197.431360887" Oct 31 04:09:39.362932 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:09:39.362897 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:09:44.230662 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:44.230631 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:09:47.111020 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.110979 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld"] Oct 31 04:09:47.116550 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.116519 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.128182 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.128102 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8a336e\"/\"default-dockercfg-hf4nj\"" Oct 31 04:09:47.128646 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.128593 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8a336e\"/\"simulator-single-main-8a336e-kserve-self-signed-certs\"" Oct 31 04:09:47.128971 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.128927 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8a336e\"/\"kube-root-ca.crt\"" Oct 31 04:09:47.129096 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.129082 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8a336e\"/\"openshift-service-ca.crt\"" Oct 31 04:09:47.132474 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.132453 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld"] Oct 31 04:09:47.158232 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.158172 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-model-cache\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.158409 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.158306 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-tmp-dir\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.158409 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.158339 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50951793-7f83-4563-aec7-857e3e58bf14-tls-certs\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.158409 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.158377 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfl7\" (UniqueName: \"kubernetes.io/projected/50951793-7f83-4563-aec7-857e3e58bf14-kube-api-access-7jfl7\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.158558 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.158486 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-dshm\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.158558 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.158529 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-home\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259101 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259058 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50951793-7f83-4563-aec7-857e3e58bf14-tls-certs\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259117 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfl7\" (UniqueName: \"kubernetes.io/projected/50951793-7f83-4563-aec7-857e3e58bf14-kube-api-access-7jfl7\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259146 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-dshm\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259175 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-home\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259252 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-model-cache\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259312 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259309 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-tmp-dir\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259733 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259704 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-home\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259849 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259715 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-tmp-dir\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.259849 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.259774 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-model-cache\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.261612 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.261592 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-dshm\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.261787 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.261771 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50951793-7f83-4563-aec7-857e3e58bf14-tls-certs\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.270733 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.270708 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfl7\" (UniqueName: \"kubernetes.io/projected/50951793-7f83-4563-aec7-857e3e58bf14-kube-api-access-7jfl7\") pod \"simulator-single-main-8a336e-kserve-6b479c6778-bhqld\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.435844 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.435807 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:47.710914 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:47.710837 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld"] Oct 31 04:09:47.713975 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:09:47.713945 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50951793_7f83_4563_aec7_857e3e58bf14.slice/crio-30318bade1b173cf04e21fa1867e7ec64a2de0843589537e6c8d284dacb32099 WatchSource:0}: Error finding container 30318bade1b173cf04e21fa1867e7ec64a2de0843589537e6c8d284dacb32099: Status 404 returned error can't find the container with id 30318bade1b173cf04e21fa1867e7ec64a2de0843589537e6c8d284dacb32099 Oct 31 04:09:48.305768 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:48.305718 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" event={"ID":"50951793-7f83-4563-aec7-857e3e58bf14","Type":"ContainerStarted","Data":"bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af"} Oct 31 04:09:48.306218 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:48.305778 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" event={"ID":"50951793-7f83-4563-aec7-857e3e58bf14","Type":"ContainerStarted","Data":"30318bade1b173cf04e21fa1867e7ec64a2de0843589537e6c8d284dacb32099"} Oct 31 04:09:48.324184 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:48.324125 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" podStartSLOduration=1.3241100590000001 podStartE2EDuration="1.324110059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:48.322943309 +0000 UTC m=+3212.517466431" watchObservedRunningTime="2026-10-31 04:09:48.324110059 +0000 UTC m=+3212.518633175" Oct 31 04:09:48.443259 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:48.441910 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:57.436638 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:57.436605 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:57.448852 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:57.448827 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:09:58.360060 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:09:58.360027 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:10:09.358834 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:10:09.358790 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:10:33.513866 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:33.513778 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld"] Oct 31 04:10:38.884829 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:38.884787 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5"] Oct 31 04:10:39.357375 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:10:39.357338 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:10:43.885393 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:43.885342 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" podUID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" containerName="payload-pre-processing" containerID="cri-o://f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190" gracePeriod=25 Oct 31 04:10:44.228790 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:44.228749 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" podUID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:10:48.534441 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.534363 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" podUID="50951793-7f83-4563-aec7-857e3e58bf14" containerName="main" containerID="cri-o://bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af" gracePeriod=45 Oct 31 04:10:48.775151 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.775126 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:10:48.896485 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896394 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-model-cache\") pod \"50951793-7f83-4563-aec7-857e3e58bf14\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " Oct 31 04:10:48.896485 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896444 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-home\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-home\") pod \"50951793-7f83-4563-aec7-857e3e58bf14\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " Oct 31 04:10:48.896747 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896545 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-dshm\") pod \"50951793-7f83-4563-aec7-857e3e58bf14\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " Oct 31 04:10:48.896747 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896631 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50951793-7f83-4563-aec7-857e3e58bf14-kube-api-access-7jfl7\" (UniqueName: \"kubernetes.io/projected/50951793-7f83-4563-aec7-857e3e58bf14-kube-api-access-7jfl7\") pod \"50951793-7f83-4563-aec7-857e3e58bf14\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " Oct 31 04:10:48.896747 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896658 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50951793-7f83-4563-aec7-857e3e58bf14-tls-certs\" (UniqueName: \"kubernetes.io/secret/50951793-7f83-4563-aec7-857e3e58bf14-tls-certs\") pod \"50951793-7f83-4563-aec7-857e3e58bf14\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " Oct 31 04:10:48.896747 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896710 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-tmp-dir\") pod \"50951793-7f83-4563-aec7-857e3e58bf14\" (UID: \"50951793-7f83-4563-aec7-857e3e58bf14\") " Oct 31 04:10:48.896747 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896688 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-model-cache" pod "50951793-7f83-4563-aec7-857e3e58bf14" (UID: "50951793-7f83-4563-aec7-857e3e58bf14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:48.896998 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.896762 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-home" pod "50951793-7f83-4563-aec7-857e3e58bf14" (UID: "50951793-7f83-4563-aec7-857e3e58bf14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:48.897056 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.897030 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:48.897056 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.897050 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:48.897184 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.897157 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-tmp-dir" pod "50951793-7f83-4563-aec7-857e3e58bf14" (UID: "50951793-7f83-4563-aec7-857e3e58bf14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:48.899625 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.899586 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50951793-7f83-4563-aec7-857e3e58bf14-kube-api-access-7jfl7" pod "50951793-7f83-4563-aec7-857e3e58bf14" (UID: "50951793-7f83-4563-aec7-857e3e58bf14"). InnerVolumeSpecName "kube-api-access-7jfl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:48.900862 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.900831 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50951793-7f83-4563-aec7-857e3e58bf14-tls-certs" pod "50951793-7f83-4563-aec7-857e3e58bf14" (UID: "50951793-7f83-4563-aec7-857e3e58bf14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:48.901088 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.900933 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-dshm" pod "50951793-7f83-4563-aec7-857e3e58bf14" (UID: "50951793-7f83-4563-aec7-857e3e58bf14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:48.998158 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.998121 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfl7\" (UniqueName: \"kubernetes.io/projected/50951793-7f83-4563-aec7-857e3e58bf14-kube-api-access-7jfl7\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:48.998158 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.998152 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50951793-7f83-4563-aec7-857e3e58bf14-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:48.998158 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.998161 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:48.998446 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:48.998169 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50951793-7f83-4563-aec7-857e3e58bf14-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:49.611130 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.611090 5520 generic.go:356] "Generic (PLEG): container finished" podID="50951793-7f83-4563-aec7-857e3e58bf14" containerID="bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af" exitCode=0 Oct 31 04:10:49.611739 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.611140 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" Oct 31 04:10:49.611739 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.611148 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" event={"ID":"50951793-7f83-4563-aec7-857e3e58bf14","Type":"ContainerDied","Data":"bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af"} Oct 31 04:10:49.611739 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.611188 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld" event={"ID":"50951793-7f83-4563-aec7-857e3e58bf14","Type":"ContainerDied","Data":"30318bade1b173cf04e21fa1867e7ec64a2de0843589537e6c8d284dacb32099"} Oct 31 04:10:49.611739 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.611231 5520 scope.go:122] "RemoveContainer" containerID="bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af" Oct 31 04:10:49.621527 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.621507 5520 scope.go:122] "RemoveContainer" containerID="bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af" Oct 31 04:10:49.621824 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:10:49.621804 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af\": container with ID starting with bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af not found: ID does not exist" containerID="bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af" Oct 31 04:10:49.621879 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.621834 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af"} err="failed to get container status \"bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af\": rpc error: code = NotFound desc = could not find container \"bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af\": container with ID starting with bfa3164f4bfbddf13119cec12d100d3799ab1d4803514e93fa2d27d7b75310af not found: ID does not exist" Oct 31 04:10:49.631893 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.631861 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld"] Oct 31 04:10:49.635691 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.635668 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-8a336e/simulator-single-main-8a336e-kserve-6b479c6778-bhqld"] Oct 31 04:10:49.918381 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:49.918306 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50951793-7f83-4563-aec7-857e3e58bf14" path="/var/lib/kubelet/pods/50951793-7f83-4563-aec7-857e3e58bf14/volumes" Oct 31 04:10:51.555805 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.555740 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4"] Oct 31 04:10:51.556229 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.556146 5520 cpu_manager.go:414] "containerMap: removing container" podUID="50951793-7f83-4563-aec7-857e3e58bf14" containerName="main" Oct 31 04:10:51.556229 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.556158 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50951793-7f83-4563-aec7-857e3e58bf14" containerName="main" Oct 31 04:10:51.556316 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.556252 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="50951793-7f83-4563-aec7-857e3e58bf14" containerName="main" Oct 31 04:10:51.560630 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.560603 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.571380 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.571355 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4"] Oct 31 04:10:51.571679 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.571657 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tkb4s\"" Oct 31 04:10:51.571786 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.571740 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 04:10:51.571864 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.571851 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 04:10:51.572003 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.571968 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-85f4c8a5-kserve-self-signed-certs\"" Oct 31 04:10:51.725990 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.725954 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-tmp-dir\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.726195 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.725995 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-home\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.726195 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.726018 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-dshm\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.726195 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.726128 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05992420-7c61-4c95-b806-e88fb37a72c8-tls-certs\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.726195 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.726179 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w578s\" (UniqueName: \"kubernetes.io/projected/05992420-7c61-4c95-b806-e88fb37a72c8-kube-api-access-w578s\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.726388 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.726262 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-model-cache\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827471 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827377 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-model-cache\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827654 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827479 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-tmp-dir\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827654 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827512 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-home\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827654 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827538 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-dshm\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827654 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827570 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05992420-7c61-4c95-b806-e88fb37a72c8-tls-certs\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827654 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827620 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w578s\" (UniqueName: \"kubernetes.io/projected/05992420-7c61-4c95-b806-e88fb37a72c8-kube-api-access-w578s\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827931 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827873 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-model-cache\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.827931 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.827885 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-tmp-dir\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.828056 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.828024 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-home\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.830094 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.830063 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-dshm\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.830266 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.830250 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05992420-7c61-4c95-b806-e88fb37a72c8-tls-certs\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.835266 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.835243 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w578s\" (UniqueName: \"kubernetes.io/projected/05992420-7c61-4c95-b806-e88fb37a72c8-kube-api-access-w578s\") pod \"e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:51.877532 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:51.877492 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:52.152819 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:52.152789 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4"] Oct 31 04:10:52.155881 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:10:52.155853 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05992420_7c61_4c95_b806_e88fb37a72c8.slice/crio-413b8935c68bbccf6247cc4a6f3e779d7a896a1a34351eab22471a64b8eb8459 WatchSource:0}: Error finding container 413b8935c68bbccf6247cc4a6f3e779d7a896a1a34351eab22471a64b8eb8459: Status 404 returned error can't find the container with id 413b8935c68bbccf6247cc4a6f3e779d7a896a1a34351eab22471a64b8eb8459 Oct 31 04:10:52.628853 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:52.628814 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" event={"ID":"05992420-7c61-4c95-b806-e88fb37a72c8","Type":"ContainerStarted","Data":"ee399c0e29d1fb1e8d5993b47ac6792144d5f12e46f021048a9c8b925ea07385"} Oct 31 04:10:52.629264 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:52.628863 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" event={"ID":"05992420-7c61-4c95-b806-e88fb37a72c8","Type":"ContainerStarted","Data":"413b8935c68bbccf6247cc4a6f3e779d7a896a1a34351eab22471a64b8eb8459"} Oct 31 04:10:52.645754 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:52.645696 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" podStartSLOduration=1.645677576 podStartE2EDuration="1.645677576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:52.644045821 +0000 UTC m=+3276.838568957" watchObservedRunningTime="2026-10-31 04:10:52.645677576 +0000 UTC m=+3276.840200703" Oct 31 04:10:52.878499 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:52.878460 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:10:54.229459 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:54.229415 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" podUID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:10:55.822227 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:55.821319 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kzdch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:10:55.822227 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:55.821389 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kzdch" podUID="6532e670-a319-423c-8f6f-ccea217ec5a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:10:55.866750 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:55.866689 5520 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bdwhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:10:55.867084 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:10:55.867032 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bdwhb" podUID="21db5238-717b-480c-ab5a-c1d81658a00e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:11:01.878058 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:01.878017 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:11:01.890756 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:01.890723 5520 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:11:02.703760 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:02.703731 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:11:04.124675 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.124649 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:11:04.285316 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.285290 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-kube-api-access-vrpnn\" (UniqueName: \"kubernetes.io/projected/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-kube-api-access-vrpnn\") pod \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " Oct 31 04:11:04.285509 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.285347 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-plugins-config-volume\") pod \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\" (UID: \"ac7e466e-e1ee-439d-ac49-29ccb3bd7573\") " Oct 31 04:11:04.285728 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.285704 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-plugins-config-volume" pod "ac7e466e-e1ee-439d-ac49-29ccb3bd7573" (UID: "ac7e466e-e1ee-439d-ac49-29ccb3bd7573"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:04.287590 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.287561 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-kube-api-access-vrpnn" pod "ac7e466e-e1ee-439d-ac49-29ccb3bd7573" (UID: "ac7e466e-e1ee-439d-ac49-29ccb3bd7573"). InnerVolumeSpecName "kube-api-access-vrpnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:04.386923 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.386886 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpnn\" (UniqueName: \"kubernetes.io/projected/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-kube-api-access-vrpnn\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:04.386923 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.386917 5520 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7e466e-e1ee-439d-ac49-29ccb3bd7573-plugins-config-volume\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:04.702783 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.702749 5520 generic.go:356] "Generic (PLEG): container finished" podID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" containerID="f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190" exitCode=0 Oct 31 04:11:04.702961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.702793 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" Oct 31 04:11:04.702961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.702835 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" event={"ID":"ac7e466e-e1ee-439d-ac49-29ccb3bd7573","Type":"ContainerDied","Data":"f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190"} Oct 31 04:11:04.702961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.702887 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5" event={"ID":"ac7e466e-e1ee-439d-ac49-29ccb3bd7573","Type":"ContainerDied","Data":"87f0b771924e728420d17c0925f2c299a93ced7143c580faef029bdc82ccb062"} Oct 31 04:11:04.702961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.702908 5520 scope.go:122] "RemoveContainer" containerID="f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190" Oct 31 04:11:04.712302 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.712243 5520 scope.go:122] "RemoveContainer" containerID="f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190" Oct 31 04:11:04.712543 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:11:04.712505 5520 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190\": container with ID starting with f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190 not found: ID does not exist" containerID="f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190" Oct 31 04:11:04.712543 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.712531 5520 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190"} err="failed to get container status \"f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190\": rpc error: code = NotFound desc = could not find container \"f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190\": container with ID starting with f1e28937ec986ebb1c30c9d1e49b60108a65890dff63a192e9e50fc7266b0190 not found: ID does not exist" Oct 31 04:11:04.724779 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.724759 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5"] Oct 31 04:11:04.728358 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:04.728334 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8a336e-676ccl47l5"] Oct 31 04:11:05.915112 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:05.915068 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" path="/var/lib/kubelet/pods/ac7e466e-e1ee-439d-ac49-29ccb3bd7573/volumes" Oct 31 04:11:09.363092 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:11:09.363046 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:11:35.500075 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:35.500024 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4"] Oct 31 04:11:39.357885 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:11:39.357841 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:11:48.021922 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:48.021878 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5"] Oct 31 04:11:48.024231 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:48.024177 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8447e295-kcch5"] Oct 31 04:11:49.951231 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:49.949285 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b08e78-5e6b-4a0a-a0d8-afa25756816c" path="/var/lib/kubelet/pods/32b08e78-5e6b-4a0a-a0d8-afa25756816c/volumes" Oct 31 04:11:50.537988 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:50.537931 5520 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" podUID="05992420-7c61-4c95-b806-e88fb37a72c8" containerName="main" containerID="cri-o://ee399c0e29d1fb1e8d5993b47ac6792144d5f12e46f021048a9c8b925ea07385" gracePeriod=45 Oct 31 04:11:50.696999 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:50.696642 5520 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" podUID="05992420-7c61-4c95-b806-e88fb37a72c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.255:8000/ready\": dial tcp 10.134.0.255:8000: connect: connection refused" Oct 31 04:11:51.020537 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.020349 5520 generic.go:356] "Generic (PLEG): container finished" podID="05992420-7c61-4c95-b806-e88fb37a72c8" containerID="ee399c0e29d1fb1e8d5993b47ac6792144d5f12e46f021048a9c8b925ea07385" exitCode=0 Oct 31 04:11:51.020537 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.020406 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" event={"ID":"05992420-7c61-4c95-b806-e88fb37a72c8","Type":"ContainerDied","Data":"ee399c0e29d1fb1e8d5993b47ac6792144d5f12e46f021048a9c8b925ea07385"} Oct 31 04:11:51.068937 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.068909 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:11:51.169514 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169483 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-tmp-dir\") pod \"05992420-7c61-4c95-b806-e88fb37a72c8\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " Oct 31 04:11:51.169729 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169526 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-home\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-home\") pod \"05992420-7c61-4c95-b806-e88fb37a72c8\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " Oct 31 04:11:51.169729 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169560 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05992420-7c61-4c95-b806-e88fb37a72c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/05992420-7c61-4c95-b806-e88fb37a72c8-tls-certs\") pod \"05992420-7c61-4c95-b806-e88fb37a72c8\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " Oct 31 04:11:51.169729 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169616 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-model-cache\") pod \"05992420-7c61-4c95-b806-e88fb37a72c8\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " Oct 31 04:11:51.169729 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169664 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-dshm\") pod \"05992420-7c61-4c95-b806-e88fb37a72c8\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " Oct 31 04:11:51.169729 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169719 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05992420-7c61-4c95-b806-e88fb37a72c8-kube-api-access-w578s\" (UniqueName: \"kubernetes.io/projected/05992420-7c61-4c95-b806-e88fb37a72c8-kube-api-access-w578s\") pod \"05992420-7c61-4c95-b806-e88fb37a72c8\" (UID: \"05992420-7c61-4c95-b806-e88fb37a72c8\") " Oct 31 04:11:51.170021 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169791 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-tmp-dir" pod "05992420-7c61-4c95-b806-e88fb37a72c8" (UID: "05992420-7c61-4c95-b806-e88fb37a72c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.170021 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169826 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-home" pod "05992420-7c61-4c95-b806-e88fb37a72c8" (UID: "05992420-7c61-4c95-b806-e88fb37a72c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.170021 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.169894 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-model-cache" pod "05992420-7c61-4c95-b806-e88fb37a72c8" (UID: "05992420-7c61-4c95-b806-e88fb37a72c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.170169 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.170061 5520 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-tmp-dir\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.170169 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.170080 5520 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-home\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.170169 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.170093 5520 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-model-cache\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.172007 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.171970 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-dshm" pod "05992420-7c61-4c95-b806-e88fb37a72c8" (UID: "05992420-7c61-4c95-b806-e88fb37a72c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.172136 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.172050 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05992420-7c61-4c95-b806-e88fb37a72c8-tls-certs" pod "05992420-7c61-4c95-b806-e88fb37a72c8" (UID: "05992420-7c61-4c95-b806-e88fb37a72c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:51.172136 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.172059 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05992420-7c61-4c95-b806-e88fb37a72c8-kube-api-access-w578s" pod "05992420-7c61-4c95-b806-e88fb37a72c8" (UID: "05992420-7c61-4c95-b806-e88fb37a72c8"). InnerVolumeSpecName "kube-api-access-w578s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:51.270648 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.270565 5520 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05992420-7c61-4c95-b806-e88fb37a72c8-tls-certs\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.270648 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.270592 5520 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05992420-7c61-4c95-b806-e88fb37a72c8-dshm\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.270648 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:51.270602 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w578s\" (UniqueName: \"kubernetes.io/projected/05992420-7c61-4c95-b806-e88fb37a72c8-kube-api-access-w578s\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:52.028407 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:52.028365 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" Oct 31 04:11:52.028407 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:52.028381 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4" event={"ID":"05992420-7c61-4c95-b806-e88fb37a72c8","Type":"ContainerDied","Data":"413b8935c68bbccf6247cc4a6f3e779d7a896a1a34351eab22471a64b8eb8459"} Oct 31 04:11:52.029015 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:52.028452 5520 scope.go:122] "RemoveContainer" containerID="ee399c0e29d1fb1e8d5993b47ac6792144d5f12e46f021048a9c8b925ea07385" Oct 31 04:11:52.047531 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:52.047473 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4"] Oct 31 04:11:52.050855 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:52.050825 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-85f4c8a5-kserve-5d9f84fb9-cp2f4"] Oct 31 04:11:53.980733 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:53.980696 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05992420-7c61-4c95-b806-e88fb37a72c8" path="/var/lib/kubelet/pods/05992420-7c61-4c95-b806-e88fb37a72c8/volumes" Oct 31 04:11:58.094943 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.094859 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j"] Oct 31 04:11:58.095404 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.095239 5520 cpu_manager.go:414] "containerMap: removing container" podUID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" containerName="payload-pre-processing" Oct 31 04:11:58.095404 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.095251 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" containerName="payload-pre-processing" Oct 31 04:11:58.095404 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.095269 5520 cpu_manager.go:414] "containerMap: removing container" podUID="05992420-7c61-4c95-b806-e88fb37a72c8" containerName="main" Oct 31 04:11:58.095404 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.095275 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05992420-7c61-4c95-b806-e88fb37a72c8" containerName="main" Oct 31 04:11:58.095404 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.095330 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7e466e-e1ee-439d-ac49-29ccb3bd7573" containerName="payload-pre-processing" Oct 31 04:11:58.095404 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.095340 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="05992420-7c61-4c95-b806-e88fb37a72c8" containerName="main" Oct 31 04:11:58.099439 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.099419 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.105540 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.105516 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:11:58.105762 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.105555 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:11:58.105843 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.105556 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:11:58.105843 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.105617 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\"" Oct 31 04:11:58.106838 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.106818 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j"] Oct 31 04:11:58.188832 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.188794 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvc5r\" (UniqueName: \"kubernetes.io/projected/5145cd99-5571-41d9-b8bd-e5d76400bc36-kube-api-access-lvc5r\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.189027 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.188919 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.189027 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.188985 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.290496 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.290458 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.290680 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.290514 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.290680 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.290541 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvc5r\" (UniqueName: \"kubernetes.io/projected/5145cd99-5571-41d9-b8bd-e5d76400bc36-kube-api-access-lvc5r\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.291111 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.291089 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.292971 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.292953 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.299431 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.299407 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvc5r\" (UniqueName: \"kubernetes.io/projected/5145cd99-5571-41d9-b8bd-e5d76400bc36-kube-api-access-lvc5r\") pod \"maas-api-revoke-sub-543ba813-tt44j\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.415110 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.415020 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:11:58.689249 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:58.689194 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j"] Oct 31 04:11:58.691697 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:11:58.691662 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5145cd99_5571_41d9_b8bd_e5d76400bc36.slice/crio-d775cdfdb61e757ec516a3221668aff162827ea04c1ecb1bd5bc24ac5f7cb7cf WatchSource:0}: Error finding container d775cdfdb61e757ec516a3221668aff162827ea04c1ecb1bd5bc24ac5f7cb7cf: Status 404 returned error can't find the container with id d775cdfdb61e757ec516a3221668aff162827ea04c1ecb1bd5bc24ac5f7cb7cf Oct 31 04:11:59.084517 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:59.084477 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" event={"ID":"5145cd99-5571-41d9-b8bd-e5d76400bc36","Type":"ContainerStarted","Data":"b53e95d621de5cc4fd43e5d22c87e6702b71772e2d0f4972b78594ec60028ac8"} Oct 31 04:11:59.084704 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:59.084532 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" event={"ID":"5145cd99-5571-41d9-b8bd-e5d76400bc36","Type":"ContainerStarted","Data":"d775cdfdb61e757ec516a3221668aff162827ea04c1ecb1bd5bc24ac5f7cb7cf"} Oct 31 04:11:59.100019 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:11:59.099964 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" podStartSLOduration=1.099950354 podStartE2EDuration="1.099950354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:59.099556181 +0000 UTC m=+3343.294079317" watchObservedRunningTime="2026-10-31 04:11:59.099950354 +0000 UTC m=+3343.294473469" Oct 31 04:12:00.091195 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:00.091155 5520 generic.go:356] "Generic (PLEG): container finished" podID="5145cd99-5571-41d9-b8bd-e5d76400bc36" containerID="b53e95d621de5cc4fd43e5d22c87e6702b71772e2d0f4972b78594ec60028ac8" exitCode=0 Oct 31 04:12:00.091401 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:00.091233 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" event={"ID":"5145cd99-5571-41d9-b8bd-e5d76400bc36","Type":"ContainerDied","Data":"b53e95d621de5cc4fd43e5d22c87e6702b71772e2d0f4972b78594ec60028ac8"} Oct 31 04:12:01.222858 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.222829 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:12:01.316921 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.316875 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5145cd99-5571-41d9-b8bd-e5d76400bc36-kube-api-access-lvc5r\" (UniqueName: \"kubernetes.io/projected/5145cd99-5571-41d9-b8bd-e5d76400bc36-kube-api-access-lvc5r\") pod \"5145cd99-5571-41d9-b8bd-e5d76400bc36\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " Oct 31 04:12:01.317131 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.316949 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-xks-ca\") pod \"5145cd99-5571-41d9-b8bd-e5d76400bc36\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " Oct 31 04:12:01.317131 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.317087 5520 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-openshift-ca\") pod \"5145cd99-5571-41d9-b8bd-e5d76400bc36\" (UID: \"5145cd99-5571-41d9-b8bd-e5d76400bc36\") " Oct 31 04:12:01.317468 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.317439 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-openshift-ca" pod "5145cd99-5571-41d9-b8bd-e5d76400bc36" (UID: "5145cd99-5571-41d9-b8bd-e5d76400bc36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:12:01.319172 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.319151 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-xks-ca" pod "5145cd99-5571-41d9-b8bd-e5d76400bc36" (UID: "5145cd99-5571-41d9-b8bd-e5d76400bc36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:01.319255 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.319231 5520 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5145cd99-5571-41d9-b8bd-e5d76400bc36-kube-api-access-lvc5r" pod "5145cd99-5571-41d9-b8bd-e5d76400bc36" (UID: "5145cd99-5571-41d9-b8bd-e5d76400bc36"). InnerVolumeSpecName "kube-api-access-lvc5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:01.418412 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.418324 5520 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvc5r\" (UniqueName: \"kubernetes.io/projected/5145cd99-5571-41d9-b8bd-e5d76400bc36-kube-api-access-lvc5r\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:01.418412 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.418357 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-xks-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:01.418412 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:01.418367 5520 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5145cd99-5571-41d9-b8bd-e5d76400bc36-maas-api-openshift-ca\") on node \"ip-10-0-141-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:02.101918 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:02.101882 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" event={"ID":"5145cd99-5571-41d9-b8bd-e5d76400bc36","Type":"ContainerDied","Data":"d775cdfdb61e757ec516a3221668aff162827ea04c1ecb1bd5bc24ac5f7cb7cf"} Oct 31 04:12:02.101918 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:02.101906 5520 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j" Oct 31 04:12:02.102151 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:02.101931 5520 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d775cdfdb61e757ec516a3221668aff162827ea04c1ecb1bd5bc24ac5f7cb7cf" Oct 31 04:12:09.370415 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:12:09.370369 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:12:17.761996 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.761959 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bf8887774-8pvkd"] Oct 31 04:12:17.762418 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.762329 5520 cpu_manager.go:414] "containerMap: removing container" podUID="5145cd99-5571-41d9-b8bd-e5d76400bc36" containerName="revoke-keys" Oct 31 04:12:17.762418 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.762341 5520 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5145cd99-5571-41d9-b8bd-e5d76400bc36" containerName="revoke-keys" Oct 31 04:12:17.762503 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.762422 5520 memory_manager.go:365] "RemoveStaleState removing state" podUID="5145cd99-5571-41d9-b8bd-e5d76400bc36" containerName="revoke-keys" Oct 31 04:12:17.765403 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.765383 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.772184 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.772166 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 04:12:17.772289 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.772275 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbwgh\"" Oct 31 04:12:17.773695 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.773671 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 04:12:17.773890 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.773856 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 04:12:17.775578 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.775555 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bf8887774-8pvkd"] Oct 31 04:12:17.778189 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.778167 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-metrics-certs\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.778357 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.778330 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-maas-api-tls\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.778421 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.778403 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-openshift-service-ca\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.778490 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.778473 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4md\" (UniqueName: \"kubernetes.io/projected/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-kube-api-access-km4md\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.870615 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.870577 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 04:12:17.879069 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.879036 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-metrics-certs\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.879166 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.879117 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-maas-api-tls\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.879166 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.879134 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-openshift-service-ca\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.879166 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.879161 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4md\" (UniqueName: \"kubernetes.io/projected/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-kube-api-access-km4md\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.879815 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.879792 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-openshift-service-ca\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.881758 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.881735 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-maas-api-tls\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.881860 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.881770 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-metrics-certs\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:17.889984 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:17.889946 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4md\" (UniqueName: \"kubernetes.io/projected/ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5-kube-api-access-km4md\") pod \"maas-api-bf8887774-8pvkd\" (UID: \"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:18.079423 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:18.079332 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:18.355497 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:18.355418 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bf8887774-8pvkd"] Oct 31 04:12:18.358359 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:12:18.358330 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2a1f83_b4b7_4409_8ab8_9bc5f1db03e5.slice/crio-806da74ff8f959a77c46c6dfbfef4ed21416ff14a5887c8e3a6b53aa22629875 WatchSource:0}: Error finding container 806da74ff8f959a77c46c6dfbfef4ed21416ff14a5887c8e3a6b53aa22629875: Status 404 returned error can't find the container with id 806da74ff8f959a77c46c6dfbfef4ed21416ff14a5887c8e3a6b53aa22629875 Oct 31 04:12:18.359599 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:18.359579 5520 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:12:19.179017 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:19.178983 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" event={"ID":"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5","Type":"ContainerStarted","Data":"897f2208211447a9fc0f54e55cf36fd5e4267ae3d7f0c7c10f88bda27deccbdd"} Oct 31 04:12:19.179440 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:19.179041 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" event={"ID":"ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5","Type":"ContainerStarted","Data":"806da74ff8f959a77c46c6dfbfef4ed21416ff14a5887c8e3a6b53aa22629875"} Oct 31 04:12:19.179440 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:19.179172 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:19.198008 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:19.197951 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" podStartSLOduration=1.80615154 podStartE2EDuration="2.197937829s" totalImagesPullingTime="391.786289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:19.195940548 +0000 UTC m=+3363.390463683" watchObservedRunningTime="2026-10-31 04:12:19.197937829 +0000 UTC m=+3363.392460949" Oct 31 04:12:25.189376 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:12:25.189344 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bf8887774-8pvkd" Oct 31 04:12:39.357907 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:12:39.357864 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:13:09.358183 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:13:09.358137 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:13:19.210500 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:13:19.210466 5520 scope.go:122] "RemoveContainer" containerID="f9dbec417ab2bcee8f555ea67353b678ba884090af7b5397ada2128186aedfac" Oct 31 04:13:19.218775 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:13:19.218749 5520 scope.go:122] "RemoveContainer" containerID="ab713b6524fade17aed28a5adca92fd96e940ae5d6f7b5f97f4c6d91bf801aeb" Oct 31 04:13:39.358066 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:13:39.357981 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:14:00.022762 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:14:00.022724 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg"] Oct 31 04:14:00.026874 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:14:00.026845 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-w5vvg"] Oct 31 04:14:01.914727 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:14:01.914695 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e88f23-618c-4c1f-a8ac-3f8dd36a20b1" path="/var/lib/kubelet/pods/87e88f23-618c-4c1f-a8ac-3f8dd36a20b1/volumes" Oct 31 04:14:09.356571 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:14:09.356529 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:14:19.018961 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:14:19.018922 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md"] Oct 31 04:14:19.020941 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:14:19.020918 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-gh5md"] Oct 31 04:14:19.914952 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:14:19.914919 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd39867c-bb78-4c15-b88a-5576789791a0" path="/var/lib/kubelet/pods/bd39867c-bb78-4c15-b88a-5576789791a0/volumes" Oct 31 04:14:39.357413 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:14:39.357377 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:15:09.357566 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:15:09.357523 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:15:19.252665 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:19.252631 5520 scope.go:122] "RemoveContainer" containerID="d3af41c097c8215739a1ee2ea6c2a8dcb4877d343ac03151c1075b8a8bb02e8f" Oct 31 04:15:19.261508 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:19.261479 5520 scope.go:122] "RemoveContainer" containerID="b4baa350bb172b611d0e9b9e70cd5e5bec60461a0705694fb96c11686c5e7c1f" Oct 31 04:15:39.357234 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:15:39.357177 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:15:51.782941 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:51.782910 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ntrp9_b4ce577b-f8f3-40d6-ba23-6b57cbfcea65/manager/0.log" Oct 31 04:15:52.228610 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:52.228581 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s22vp_1442a889-201b-44a2-ac6a-b9d37d5f81fe/server/0.log" Oct 31 04:15:53.868035 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:53.867970 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bf8887774-8pvkd_ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5/maas-api/0.log" Oct 31 04:15:54.764451 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:54.764424 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-543ba813-tt44j_5145cd99-5571-41d9-b8bd-e5d76400bc36/revoke-keys/0.log" Oct 31 04:15:57.253096 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:57.253069 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8xbmg_8cf2112a-59fe-42b2-923d-3bd8e988fa05/kuadrant-console-plugin/0.log" Oct 31 04:15:58.966953 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:58.966923 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-w62v8_6afccb7d-3c0d-4953-9b3b-2f912d6e4b34/discovery/0.log" Oct 31 04:15:59.335259 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:59.335159 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-zt5k5_8aec38a2-ea3c-4bc3-a971-10953e2299e5/kube-auth-proxy/0.log" Oct 31 04:15:59.514620 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:15:59.514587 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-5wkxx_739ca5c0-6636-4c97-9236-04a5af2564d6/istio-proxy/0.log" Oct 31 04:16:00.042930 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:00.042895 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-bdwhb_21db5238-717b-480c-ab5a-c1d81658a00e/istio-proxy/0.log" Oct 31 04:16:00.217988 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:00.217952 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-f6mgl_91077eb7-9c1a-4363-bdbf-e65fac8395d8/istio-proxy/0.log" Oct 31 04:16:00.389062 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:00.388988 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-kzdch_6532e670-a319-423c-8f6f-ccea217ec5a2/istio-proxy/0.log" Oct 31 04:16:09.357721 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:16:09.357673 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:16:13.774470 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:13.774433 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ntrp9_b4ce577b-f8f3-40d6-ba23-6b57cbfcea65/manager/0.log" Oct 31 04:16:14.228491 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:14.228465 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-s22vp_1442a889-201b-44a2-ac6a-b9d37d5f81fe/server/0.log" Oct 31 04:16:15.119872 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:15.119827 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bf8887774-8pvkd_ad2a1f83-b4b7-4409-8ab8-9bc5f1db03e5/maas-api/0.log" Oct 31 04:16:16.017147 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:16.017113 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-543ba813-tt44j_5145cd99-5571-41d9-b8bd-e5d76400bc36/revoke-keys/0.log" Oct 31 04:16:18.249369 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:18.249339 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8xbmg_8cf2112a-59fe-42b2-923d-3bd8e988fa05/kuadrant-console-plugin/0.log" Oct 31 04:16:19.539947 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:19.539915 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-w62v8_6afccb7d-3c0d-4953-9b3b-2f912d6e4b34/discovery/0.log" Oct 31 04:16:19.918975 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:19.918896 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-zt5k5_8aec38a2-ea3c-4bc3-a971-10953e2299e5/kube-auth-proxy/0.log" Oct 31 04:16:20.093917 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:20.093890 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-5wkxx_739ca5c0-6636-4c97-9236-04a5af2564d6/istio-proxy/0.log" Oct 31 04:16:20.649860 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:20.649824 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-bdwhb_21db5238-717b-480c-ab5a-c1d81658a00e/istio-proxy/0.log" Oct 31 04:16:20.825610 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:20.825581 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-f6mgl_91077eb7-9c1a-4363-bdbf-e65fac8395d8/istio-proxy/0.log" Oct 31 04:16:21.009672 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:16:21.009636 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-kzdch_6532e670-a319-423c-8f6f-ccea217ec5a2/istio-proxy/0.log" Oct 31 04:16:39.357948 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:16:39.357904 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:17:02.020821 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:02.020781 5520 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j"] Oct 31 04:17:02.022675 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:02.022650 5520 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543ba813-tt44j"] Oct 31 04:17:03.915728 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:03.915693 5520 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5145cd99-5571-41d9-b8bd-e5d76400bc36" path="/var/lib/kubelet/pods/5145cd99-5571-41d9-b8bd-e5d76400bc36/volumes" Oct 31 04:17:09.357558 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:17:09.357518 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:17:39.357429 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:17:39.357388 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:17:54.261086 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.261046 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6qbd4/must-gather-2p4m8"] Oct 31 04:17:54.264390 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.264371 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.268034 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.268006 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qbd4\"/\"openshift-service-ca.crt\"" Oct 31 04:17:54.268849 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.268833 5520 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6qbd4\"/\"default-dockercfg-f7krs\"" Oct 31 04:17:54.268931 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.268910 5520 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qbd4\"/\"kube-root-ca.crt\"" Oct 31 04:17:54.280148 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.280120 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/must-gather-2p4m8"] Oct 31 04:17:54.320157 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.320127 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7af619-ed4e-434d-bae5-9977a2f5d327-must-gather-output\") pod \"must-gather-2p4m8\" (UID: \"cc7af619-ed4e-434d-bae5-9977a2f5d327\") " pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.320306 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.320284 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dz7s\" (UniqueName: \"kubernetes.io/projected/cc7af619-ed4e-434d-bae5-9977a2f5d327-kube-api-access-4dz7s\") pod \"must-gather-2p4m8\" (UID: \"cc7af619-ed4e-434d-bae5-9977a2f5d327\") " pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.420750 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.420717 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7af619-ed4e-434d-bae5-9977a2f5d327-must-gather-output\") pod \"must-gather-2p4m8\" (UID: \"cc7af619-ed4e-434d-bae5-9977a2f5d327\") " pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.420929 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.420810 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dz7s\" (UniqueName: \"kubernetes.io/projected/cc7af619-ed4e-434d-bae5-9977a2f5d327-kube-api-access-4dz7s\") pod \"must-gather-2p4m8\" (UID: \"cc7af619-ed4e-434d-bae5-9977a2f5d327\") " pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.421066 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.421045 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7af619-ed4e-434d-bae5-9977a2f5d327-must-gather-output\") pod \"must-gather-2p4m8\" (UID: \"cc7af619-ed4e-434d-bae5-9977a2f5d327\") " pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.437010 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.436982 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dz7s\" (UniqueName: \"kubernetes.io/projected/cc7af619-ed4e-434d-bae5-9977a2f5d327-kube-api-access-4dz7s\") pod \"must-gather-2p4m8\" (UID: \"cc7af619-ed4e-434d-bae5-9977a2f5d327\") " pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.578771 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.578683 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/must-gather-2p4m8" Oct 31 04:17:54.849672 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.849596 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/must-gather-2p4m8"] Oct 31 04:17:54.853140 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:17:54.853113 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc7af619_ed4e_434d_bae5_9977a2f5d327.slice/crio-12d77e7695cbb6bb526a5d89df796ae70d43c783cb6cdeeb7baf06452c5fae36 WatchSource:0}: Error finding container 12d77e7695cbb6bb526a5d89df796ae70d43c783cb6cdeeb7baf06452c5fae36: Status 404 returned error can't find the container with id 12d77e7695cbb6bb526a5d89df796ae70d43c783cb6cdeeb7baf06452c5fae36 Oct 31 04:17:54.854872 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:54.854851 5520 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:17:55.682671 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:55.682586 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/must-gather-2p4m8" event={"ID":"cc7af619-ed4e-434d-bae5-9977a2f5d327","Type":"ContainerStarted","Data":"12d77e7695cbb6bb526a5d89df796ae70d43c783cb6cdeeb7baf06452c5fae36"} Oct 31 04:17:56.689991 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:56.689949 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/must-gather-2p4m8" event={"ID":"cc7af619-ed4e-434d-bae5-9977a2f5d327","Type":"ContainerStarted","Data":"c7044cfc72c635f0840aaf5ee103f6226c1d059e3a97d4f4ca16757671cde672"} Oct 31 04:17:56.690543 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:56.690515 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/must-gather-2p4m8" event={"ID":"cc7af619-ed4e-434d-bae5-9977a2f5d327","Type":"ContainerStarted","Data":"93f879ebe7fa2f873f2b6b1433de3e1bad0b6df905b87012298fac614d1a3d9c"} Oct 31 04:17:56.706036 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:56.705976 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6qbd4/must-gather-2p4m8" podStartSLOduration=1.8536510430000002 podStartE2EDuration="2.705959156s" totalImagesPullingTime="852.308113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:56.705214307 +0000 UTC m=+3700.899737422" watchObservedRunningTime="2026-10-31 04:17:56.705959156 +0000 UTC m=+3700.900482279" Oct 31 04:17:57.327388 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:57.327360 5520 ???:1] "http: TLS handshake error from 10.0.142.76:54126: EOF" Oct 31 04:17:57.332182 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:57.332151 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9cczw_73113784-937f-492c-9ac1-1a98c59d43e6/global-pull-secret-syncer/0.log" Oct 31 04:17:57.688742 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:57.688711 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t5fpc_f454b995-5cf2-461d-b6f1-932bd4a45110/konnectivity-agent/0.log" Oct 31 04:17:57.746500 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:17:57.746460 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-153.us-east-2.compute.internal_057a5e637a257f84120786f81c645c34/haproxy/0.log" Oct 31 04:18:05.037572 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:05.037341 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8xbmg_8cf2112a-59fe-42b2-923d-3bd8e988fa05/kuadrant-console-plugin/0.log" Oct 31 04:18:06.510172 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.510053 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c55d174-300d-40ef-ab2a-b8a6e5364b56/alertmanager/0.log" Oct 31 04:18:06.535192 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.535157 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c55d174-300d-40ef-ab2a-b8a6e5364b56/config-reloader/0.log" Oct 31 04:18:06.559463 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.559404 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c55d174-300d-40ef-ab2a-b8a6e5364b56/kube-rbac-proxy-web/0.log" Oct 31 04:18:06.584458 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.584420 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c55d174-300d-40ef-ab2a-b8a6e5364b56/kube-rbac-proxy/0.log" Oct 31 04:18:06.609572 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.609540 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c55d174-300d-40ef-ab2a-b8a6e5364b56/kube-rbac-proxy-metric/0.log" Oct 31 04:18:06.637451 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.637417 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c55d174-300d-40ef-ab2a-b8a6e5364b56/prom-label-proxy/0.log" Oct 31 04:18:06.667761 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.667120 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c55d174-300d-40ef-ab2a-b8a6e5364b56/init-config-reloader/0.log" Oct 31 04:18:06.762531 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.762451 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r4j9j_0577b36e-dd61-41bb-899d-c7af7b388696/kube-state-metrics/0.log" Oct 31 04:18:06.781652 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.781615 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r4j9j_0577b36e-dd61-41bb-899d-c7af7b388696/kube-rbac-proxy-main/0.log" Oct 31 04:18:06.809184 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.809158 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-r4j9j_0577b36e-dd61-41bb-899d-c7af7b388696/kube-rbac-proxy-self/0.log" Oct 31 04:18:06.836379 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.836347 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d97946f89-wfdp6_eb5cd4aa-cca0-44d0-b314-05ed0788d784/metrics-server/0.log" Oct 31 04:18:06.971786 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:06.971742 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gsvct_85ec7f68-370c-4e14-a990-877b350fcdf8/node-exporter/0.log" Oct 31 04:18:07.003668 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.003623 5520 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs"] Oct 31 04:18:07.006516 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.006493 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gsvct_85ec7f68-370c-4e14-a990-877b350fcdf8/kube-rbac-proxy/0.log" Oct 31 04:18:07.013660 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.013588 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.015120 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.015088 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs"] Oct 31 04:18:07.038717 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.038693 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gsvct_85ec7f68-370c-4e14-a990-877b350fcdf8/init-textfile/0.log" Oct 31 04:18:07.071835 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.071798 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-proc\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.072035 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.071931 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-podres\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.072119 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.072051 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-sys\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.072183 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.072164 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-lib-modules\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.072267 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.072214 5520 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kswp\" (UniqueName: \"kubernetes.io/projected/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-kube-api-access-2kswp\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.154454 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.154422 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6lkfz_75f26e8e-228a-4b68-8851-c580c827e242/kube-rbac-proxy-main/0.log" Oct 31 04:18:07.172749 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.172718 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-sys\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.172936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.172802 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-lib-modules\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.172936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.172830 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kswp\" (UniqueName: \"kubernetes.io/projected/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-kube-api-access-2kswp\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.172936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.172838 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-sys\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.172936 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.172932 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-lib-modules\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.173239 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.173003 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-proc\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.173239 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.173051 5520 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-podres\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.173239 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.173134 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-proc\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.173239 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.173181 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-podres\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.179787 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.179759 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6lkfz_75f26e8e-228a-4b68-8851-c580c827e242/kube-rbac-proxy-self/0.log" Oct 31 04:18:07.180885 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.180860 5520 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kswp\" (UniqueName: \"kubernetes.io/projected/a0298e1e-abf2-47ff-9ca2-c3c80bb95017-kube-api-access-2kswp\") pod \"perf-node-gather-daemonset-tckjs\" (UID: \"a0298e1e-abf2-47ff-9ca2-c3c80bb95017\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.203709 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.203680 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6lkfz_75f26e8e-228a-4b68-8851-c580c827e242/openshift-state-metrics/0.log" Oct 31 04:18:07.337458 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.337368 5520 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:07.511164 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.510955 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7548fffcd4-wq7fh_3c24f704-898e-406b-8e06-d70e3fb4f65a/telemeter-client/0.log" Oct 31 04:18:07.536024 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.535996 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7548fffcd4-wq7fh_3c24f704-898e-406b-8e06-d70e3fb4f65a/reload/0.log" Oct 31 04:18:07.559298 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.559259 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7548fffcd4-wq7fh_3c24f704-898e-406b-8e06-d70e3fb4f65a/kube-rbac-proxy/0.log" Oct 31 04:18:07.626294 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.624945 5520 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs"] Oct 31 04:18:07.628247 ip-10-0-141-153 kubenswrapper[5520]: W1031 04:18:07.628183 5520 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda0298e1e_abf2_47ff_9ca2_c3c80bb95017.slice/crio-432721cb7df160ec87cf868a7387601caa9984efbd04d301381ac029a3343557 WatchSource:0}: Error finding container 432721cb7df160ec87cf868a7387601caa9984efbd04d301381ac029a3343557: Status 404 returned error can't find the container with id 432721cb7df160ec87cf868a7387601caa9984efbd04d301381ac029a3343557 Oct 31 04:18:07.793570 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.793530 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" event={"ID":"a0298e1e-abf2-47ff-9ca2-c3c80bb95017","Type":"ContainerStarted","Data":"694c9126354d694397cd28c5675f9d70d58882e2a43dc4e775d96871daa6f7b5"} Oct 31 04:18:07.793928 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.793596 5520 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" event={"ID":"a0298e1e-abf2-47ff-9ca2-c3c80bb95017","Type":"ContainerStarted","Data":"432721cb7df160ec87cf868a7387601caa9984efbd04d301381ac029a3343557"} Oct 31 04:18:07.793928 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:07.793687 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:09.377103 ip-10-0-141-153 kubenswrapper[5520]: E1031 04:18:09.377064 5520 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-s8l89" containerName="extractor" Oct 31 04:18:09.866456 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:09.866427 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f6974f696-kqrj2_79b1ce9a-51ac-42a2-b4f7-08ba651898c8/console/0.log" Oct 31 04:18:11.655699 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:11.655660 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76rrr_e5deb6a1-16c9-4491-8e07-d8deafd8607f/dns/0.log" Oct 31 04:18:11.683583 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:11.683551 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76rrr_e5deb6a1-16c9-4491-8e07-d8deafd8607f/kube-rbac-proxy/0.log" Oct 31 04:18:11.807467 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:11.807433 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbm7r_3387800d-492d-4f65-b3d8-4bbccd906d03/dns-node-resolver/0.log" Oct 31 04:18:12.485697 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:12.485663 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xwkgw_196bf414-2c29-40a0-a957-f94cadaa7ff2/node-ca/0.log" Oct 31 04:18:13.813141 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:13.813113 5520 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" Oct 31 04:18:13.831411 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:13.831362 5520 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-tckjs" podStartSLOduration=7.831348854 podStartE2EDuration="7.831348854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:07.811402298 +0000 UTC m=+3712.005925422" watchObservedRunningTime="2026-10-31 04:18:13.831348854 +0000 UTC m=+3718.025871972" Oct 31 04:18:16.524507 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:16.524462 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-w62v8_6afccb7d-3c0d-4953-9b3b-2f912d6e4b34/discovery/0.log" Oct 31 04:18:17.048355 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:17.046514 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-zt5k5_8aec38a2-ea3c-4bc3-a971-10953e2299e5/kube-auth-proxy/0.log" Oct 31 04:18:17.084557 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:17.084531 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-5wkxx_739ca5c0-6636-4c97-9236-04a5af2564d6/istio-proxy/0.log" Oct 31 04:18:17.179612 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:17.179551 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-bdwhb_21db5238-717b-480c-ab5a-c1d81658a00e/istio-proxy/0.log" Oct 31 04:18:17.213132 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:17.213092 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-f6mgl_91077eb7-9c1a-4363-bdbf-e65fac8395d8/istio-proxy/0.log" Oct 31 04:18:17.247593 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:17.247563 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-kzdch_6532e670-a319-423c-8f6f-ccea217ec5a2/istio-proxy/0.log" Oct 31 04:18:19.300042 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:19.299980 5520 scope.go:122] "RemoveContainer" containerID="b53e95d621de5cc4fd43e5d22c87e6702b71772e2d0f4972b78594ec60028ac8" Oct 31 04:18:20.335176 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.335139 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vbv64_4080580a-a99a-49e0-a255-06cb08581290/serve-healthcheck-canary/0.log" Oct 31 04:18:20.766901 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.766869 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-w62v8_6afccb7d-3c0d-4953-9b3b-2f912d6e4b34/discovery/0.log" Oct 31 04:18:20.796045 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.796021 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-zt5k5_8aec38a2-ea3c-4bc3-a971-10953e2299e5/kube-auth-proxy/0.log" Oct 31 04:18:20.816485 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.816443 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-5wkxx_739ca5c0-6636-4c97-9236-04a5af2564d6/istio-proxy/0.log" Oct 31 04:18:20.873179 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.873150 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-bdwhb_21db5238-717b-480c-ab5a-c1d81658a00e/istio-proxy/0.log" Oct 31 04:18:20.889774 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.889746 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-f6mgl_91077eb7-9c1a-4363-bdbf-e65fac8395d8/istio-proxy/0.log" Oct 31 04:18:20.909892 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.909859 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-kzdch_6532e670-a319-423c-8f6f-ccea217ec5a2/istio-proxy/0.log" Oct 31 04:18:20.991361 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:20.991326 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s8l89_f09f77c2-e722-4f39-b4c9-2fcca44268a9/kube-rbac-proxy/0.log" Oct 31 04:18:21.013913 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:21.013882 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s8l89_f09f77c2-e722-4f39-b4c9-2fcca44268a9/exporter/0.log" Oct 31 04:18:21.037580 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:21.037520 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s8l89_f09f77c2-e722-4f39-b4c9-2fcca44268a9/extractor/0.log" Oct 31 04:18:23.554952 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:23.554920 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-rx6m2_f7fdbf06-a4c6-4718-b36d-6e42eceab645/manager/0.log" Oct 31 04:18:23.577903 ip-10-0-141-153 kubenswrapper[5520]: I1031 04:18:23.577876 5520 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-rx6m2_f7fdbf06-a4c6-4718-b36d-6e42eceab645/copy-manifests/0.log"